From ead79bda6444311bf887121d9104e1e544eb114c Mon Sep 17 00:00:00 2001 From: zelig Date: Tue, 11 Mar 2025 22:23:49 +0100 Subject: [PATCH 1/6] swip-35 neg incentives epic --- SWIPs/swip-35.md | 118 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 SWIPs/swip-35.md diff --git a/SWIPs/swip-35.md b/SWIPs/swip-35.md new file mode 100644 index 0000000..a9e120a --- /dev/null +++ b/SWIPs/swip-35.md @@ -0,0 +1,118 @@ +--- +SWIP: 35 +title: Feed Wrapping +author: Viktor Trón @zelig +Viktor Trón (@zelig) +discussions-to: https://discord.gg/Q6BvSkCv +dcreated: 2025-03-09 +--- + +# Advanced storage guarantees + +## Abstract + +This SWIP proposes an entire epic about adding further layers of incentives to the basic positive incentives in Swarm. By adopting the SWIP, users will be offered a wider range of constructions to secure the persistance of chunks and their metadata. + +## Motivation + +The current storage incentive system has a few shortcomings mainly related to the +problem that postage batches are difficult to interpret. + +The seemingly most natural way to pay for storage is by the amount and time period. Surely the unit price of storage (per chunk per block) already suggests this. However, for most, a deal (through an act of purchase) entails that the counterparty service can be kept to the terms precisely: i.e., you bought a fixed quota (amount of bytes) for a fixed period (duration starting now). Experience shows that any deviation from this expectation comes as a surprise for the user. + +The current primitive of storage compensation, the postage batch falls short on these primarily because it was designed with the simplicity of the intergration into the redistribution system in mind. + +The quota encoded in the batch's slots is not neutral: it also serves to force balanced use across neighbourhoods: maxing out the quota for nighbourhood, it implies maxing out the same subquota for each neighbourhood which requires no variance across NH-s; so the effective batch utilisation rate increases with overall volume, in other words there is a natural overhead for small batches[^1] + +[^1]: or more naturally put, there is wholesale bulk discount. + +Incomplete utilisation can be counteracted with making sure you can rearrange between neighbourhoods by either [mining]() or having availability assured by a constant influx of chunks (see section below). On the other hand, with the variance limited, the actual discount can also be made explicit in the current system and therefore user expectations of their quota can [bee managed](https://www.overleaf.com/4919957411cgrncysjqrmv#3b42ca). + +On the other hand, the redistribution game is a probabilistic outpayment scheme in the sense that instead of paying -- on each game all the contributors for a recurring fixed proportion of used storage we just pay out the entire reward pot for the period to a random winner at every game. This allows for central pricing sensitive to supply and can always automatically discover the optimal price if we assume a liquid provider's market. As a consequence, users do not need to speculate on the future price of storage at all, they can just trust the oracle to discover the rent price of rent: they can provide the compensation for their quota as a "hot wallet" from which the price of rent is automatically deduced. + +However, this also means that undermonetising the batch in the context of increasing rent price may lead to earlier expiry. This can only be counteracted by having more funds on the batch, so users effectively pay for security (upload and disappear experience). After the intended expiry passes, all funds remaining on the batch can be used for more storage service.[^2] + +[^2]: Using the current balance to create another batch (potenitally different size, period and owner) is currently not supported but should be. + +In what follows we argue that instead of having fixed-quota-for-fixed-period deals as a primitive, Swarm will provide these as a solution through further smart contracts on top of the postage batches. + +Surely, when users commit to a storage term, they usually have a use in mind, they want to commit to storing a particular (set of) file(s) for a period. While they understand that quota are quantised in an arbitrary way and might not match their immediate need, the surplus will often be considered waste. If users had a deal for a price for an amount for a period, leftover could be viewed as an opportunity to use the prenegotiated deal in the future. However, without such a deal, users have a hard time interpreting leftover quota on the one hand, and they expect immediate availability: a liquid market for both quota and duration. It is therefore crucial that storage providers always have available at any point, any amount for any period.[^1] + +[^1]: Instead of a continuous scale of options, period is guaranteed with an accuracy of matching expected to decrease exponentially with time. + +To sum up, upload and disappear experience with certainty can actually be guaranteed to users if storage providers themselves secure the availability of the chunks using their own postage batch. A liquid market is best provided if it is provided by a decentralised service network: the very swarm nodes themselves. However, it is best that the insurance or availability provider is not in the same neighbourhood as the chunk would normally be stored in yet the distribution is uniform. We argue that the insurance address of the chunk is the legacy Keccak hash of the chunk address H(addr). If the insurance address of the chunk falls exactly into the same neighbourhood as the address (well, the first byte match), then we fall back choosing the bitflip (bitwise XOR with the binary for $0xff$) on the first byte. This will place roughly one in every 256 chunk into the opposite side of the network which still keeps the balance of storage. + +So when nodes swear on to provide insurance/availability level X service, they register their address with a stake on a (the?) staking contract. It is important that providers should be distributed evenly in neighbourhoods, ideally of the same size as that disctated by the reserve. +Maybe all full nodes are providers by default. This is very useful because +when a node provides the service, they expect chunks to their neighbourhood by the hash of the BMT address of a CAC --- from now a new chunk type. Once they receive it, they +must provide the uploader with a signed receipt. +Some insurers may just take on the task of keeping the chunks retrieveable. For this, they need to create a postage batch matching the period requirement in the chunk request and eventually sign stamps and send the chunk with the stamp to their destination. + +They can keep the chunk in this batch retrieveable by regularly topping up the batch. Since the hashes are uniformly distributed, this batch is expected to be naturally balanced as it should be for the node to be able to realistically fill it. +The request will reference a specified expiry and the batch need to have a balance at least until that date. + +The due to the increasing price of BZZ or organic decrease of the storage price, the value may increase potentially beyond what is deduced with the rent every game. This eventuality could quietly implement Arweave's promise ("pay once, store forever"), except that you can always stop it and use it to store something else. + +These livekeepers collect and then publish chunks with their own stamp. The batch that they use will be grouping chunks by +1. H(address); locality of their node address +2. expiry; past request date +3. arrival time of request; + +As long as 3.~is undifferentiating due to a large number of (near) simultaneous requests +1 and 2 will not easily help align identity to chunk or even chunk to chunk. +This is why publishing can be delayed exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks uploaded. Problem is that if the traffic is not great, especially long term storage (more expensive), publishing may need to wait beyond the chunks of the file already disappear from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently. Surely the bigger the batch, the bigger the anonimity set, thus the better the anonimity that you get. + +In fact this opens up the possibility that livekeepers keep alive batches practically for short time periods representing a unit the others are multiples of, then they can always preserve information about how long the relevant chunks are meant to be stored. + +This construction is virtually a stamp mixer in that it blinds any originator address's association with chunks and also obfuscates the belonging together of a set of chunks. +Since association by postage batch was always the wierd anomaly, solving it is a major improvement in swarm's privacy. + + + +#### Insurer: + +#### Livekeeper-insurer: + +Overall using such a deal would allow over +gained: +even though treating the postage batch as a hot wallet has quite a few advantages (live accouting of + - *privacy*: + - effective *stamp mixing + - complete obfuscation (and plausible deniability) of chunk originator + - no leakage of shared ownership through signature by same batch owner + - *precision*: + - exact period + - exacy amount + - *predictability*: + - easier to understand pprimitives + - *preparedness*: + - real *upload and disappear* experience (no need to be around to secure) + - immediate availability of uploads for any amount/period without wait or prepurchase + - *pricing*: + - easier to understand/more pricing + - potentially complete utilisation of batches leading to cheaper prices or easier to coordinate subsidies + - prepurchase (keeping batches open) makes sense (speculation on price deal) + - moderate future price speculation needed + - 'pay once, store forever' now available to providers (only) + - no benefit from rent price drop + - *portability*: + - only providers need to keep track of batches and batch utilisation + - since users got all they need from just knowing that a file/root was insured/kept alive + + + + +## Specification + +## Rationale + +## Backward compatibility + +## Test Cases + +## Implementation + +## Copyright + + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From aa19cb1734d0bdf0695b9d549913fd8c69aa7c16 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Wed, 12 Mar 2025 03:07:40 +0100 Subject: [PATCH 2/6] Update swip-35.md headers --- SWIPs/swip-35.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/SWIPs/swip-35.md b/SWIPs/swip-35.md index a9e120a..a78862b 100644 --- a/SWIPs/swip-35.md +++ b/SWIPs/swip-35.md @@ -1,8 +1,7 @@ --- SWIP: 35 title: Feed Wrapping -author: Viktor Trón @zelig -Viktor Trón (@zelig) +author: Viktor Trón (@zelig) discussions-to: https://discord.gg/Q6BvSkCv dcreated: 2025-03-09 --- From 7e6651d5b05161f1594dc1eb508ada285f8a79f9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Wed, 12 Mar 2025 08:16:55 +0100 Subject: [PATCH 3/6] Update swip-35.md extended text,and structure still draft --- SWIPs/swip-35.md | 95 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 73 insertions(+), 22 deletions(-) diff --git a/SWIPs/swip-35.md b/SWIPs/swip-35.md index a78862b..19c4303 100644 --- a/SWIPs/swip-35.md +++ b/SWIPs/swip-35.md @@ -1,45 +1,90 @@ --- SWIP: 35 -title: Feed Wrapping +title: Negative Incentives (Epic) author: Viktor Trón (@zelig) -discussions-to: https://discord.gg/Q6BvSkCv -dcreated: 2025-03-09 +discussions-to: discord SWIP channel, https://discord.gg/Q6BvSkCv +status: Draft +type: Standards Track +category: Core +created: 2025-03-09 --- # Advanced storage guarantees ## Abstract -This SWIP proposes an entire epic about adding further layers of incentives to the basic positive incentives in Swarm. By adopting the SWIP, users will be offered a wider range of constructions to secure the persistance of chunks and their metadata. +This SWIP proposes an entire epic about adding further layers of incentives to the basic positive incentives in Swarm. By adopting the SWIP, swarm will offer a wider range of constructions for its users to secure the persistance of chunks and their metadata and at the same time allow a low barrier of entry service that node operators with extra storage or bzz future providers are able to increase their revenue with. ## Motivation -The current storage incentive system has a few shortcomings mainly related to the -problem that postage batches are difficult to interpret. +The current storage incentive system has a few shortcomings mainly related to the problem that postage batches are difficult to interpret. This interpretation problem leads to quite a bit of misunderstanding and misguided (also potentially failing) expectations regarding both the stored amount and the storage period. -The seemingly most natural way to pay for storage is by the amount and time period. Surely the unit price of storage (per chunk per block) already suggests this. However, for most, a deal (through an act of purchase) entails that the counterparty service can be kept to the terms precisely: i.e., you bought a fixed quota (amount of bytes) for a fixed period (duration starting now). Experience shows that any deviation from this expectation comes as a surprise for the user. +These issues are coupled with known privacy concerns regarding the shared ownership of chunks that are transparent to 3rd parties. Effectively information about the publisher as well as the chunks constituting the publication leak the system, somewhat overshadowing the otherwise rather strong privacy claims in swarm. -The current primitive of storage compensation, the postage batch falls short on these primarily because it was designed with the simplicity of the intergration into the redistribution system in mind. +The latter issue is solved if the stamps on individual chunks are signed by other nodes. We will introduce *stamp mixer service* of sorts in a way that the system will not lose its capacity to prevent doubly sold postage slots.[^3] -The quota encoded in the batch's slots is not neutral: it also serves to force balanced use across neighbourhoods: maxing out the quota for nighbourhood, it implies maxing out the same subquota for each neighbourhood which requires no variance across NH-s; so the effective batch utilisation rate increases with overall volume, in other words there is a natural overhead for small batches[^1] +[^3]: + +On top of all this: if batches are used one per upload session etc, bookkeeping of "what chunk to link with what batch index" is less relevant; in fact, it matters solely in order to save money in case the file is reuploaded.[^4] +In this SWIP we introduce a scheme in which uploads are not necessarily using a stamp; even though uploaders can purchase storage for arbitrary long periods. +No per-chunk metadata needs recording and made available in this case, meaning that the end-user case no longer raises portability concerns. + +Last, but not least, enforcing payments for each upload does not really make sense: firstly, users had some difficulty to grasp that given the free-tier for bandwidth inentives allowing free downlad and upload with throttled throughput, how come they need to pay for the storage. This problem was most obvious for really short-lived messages that are used in chat or notifications. Since, in these cases, preservation is not meaningful or even actively not desired, paying for postage sounds not only a unnecessary but outright wasteful. Secondly, the fact that expired chunks are not deleted but are put to the *cache*[^5] poses the question how come there is no direct entry to it. +This anomaly can be solved if we introduce free uploads, i.e., uploading chunks to Swarm without a postage stamp. + + +[^5]: so that popular content can bring bandwidth revenue even if no longer available for storage inecntive rewards. + +## Solution + +### Free uploads to cache + +Just like before, the swap protocol offers free (throttled) as well as paid (max throughput) service through cheques, now, upload also offers free (chunks are put to cache and purged if necessary) as well as paid (chunk put to the reserve until stamp expires) service using postage stamps. + + +### Stamp mixing + +#### Incentive to prevent double signing + +Postage batches primarily serve to enforce the uniform distribution of chunks, and thereby achieve a balanced load of storage across the network. As part of the solution, a user must not double spend, i.e., assign more than one chunk to the same postage slot. Originally, this is guaranteed by the fact that the owner is interested in not being caught since that may lead to the loss of their own data, i.e., data that they themselves had uploaded. If a stamp mixer of sorts is introduced, we must make sure that issuers cannot use the same slots: which happens to be the case if all chunks are insured with *negative incentives* (punitive measures). + +#### Contractual terms of storage + +The most natural way to pay for storage is by the *amount* and *time period*. Surely, the unit price of storage (per chunk, per block) already suggests this. However, for most, a deal (through an act of purchase) entails that the counterparty service can be kept to the terms *precisely*: i.e., you bought a fixed quota (amount of bytes) for a fixed period (duration starting now, expiring after a fixed period or on a fixed datte). Experience shows that any deviation from this expectation comes as a surprise for the user. + +The current primitive of storage compensation, the postage batch falls short on these. The primary reason is that it was designed with the simplicity in mind pertaining to the intergration into the redistribution system. + +##### Postage batch utilisation + +The quota encoded in the batch's slots is not neutral: as per the previous section, it also serves to force balanced use across neighbourhoods: maxing out the subquota for one neighbourhood implies maxing out the same subquota for each neighbourhood which requires no variance across NH-s; so the effective batch utilisation rate increases with overall volume, in other words there is a natural overhead for small batches[^1] [^1]: or more naturally put, there is wholesale bulk discount. -Incomplete utilisation can be counteracted with making sure you can rearrange between neighbourhoods by either [mining]() or having availability assured by a constant influx of chunks (see section below). On the other hand, with the variance limited, the actual discount can also be made explicit in the current system and therefore user expectations of their quota can [bee managed](https://www.overleaf.com/4919957411cgrncysjqrmv#3b42ca). +Incomplete utilisation can be counteracted with making sure you can rearrange between neighbourhoods by either [mining]() or having availability guaranteed by a constant influx of chunks (see section below). On the other hand, with the variance limited, the actual discount can also be made explicit in the current system and therefore [user expectations of their quota can bee managed](https://www.overleaf.com/4919957411cgrncysjqrmv#3b42ca). + +#### Storage period and expiry + +On the other hand, the redistribution game is a *probabilistic outpayment scheme* in the sense that instead of paying -- on each game all the contributors for a recurring fixed proportion of used storage we just pay out the entire reward pot for the period to a random winner at every game. This allows for central pricing sensitive to supply and offers automatic and optimal price discovery as long as we assume a liquid market of operators providing storage. As a consequence, users do not need to speculate on the future price of storage at all, they can just trust the oracle to discover the best price of rent. Users provide the compensation for their quota as a "hot wallet" from which the price of rent is automatically deduced. The postage batch is such a construct: it serves as a hot wallet. The expiry is defined as the onset of insolvency, i.e., whenever the balance goes to zero. -On the other hand, the redistribution game is a probabilistic outpayment scheme in the sense that instead of paying -- on each game all the contributors for a recurring fixed proportion of used storage we just pay out the entire reward pot for the period to a random winner at every game. This allows for central pricing sensitive to supply and can always automatically discover the optimal price if we assume a liquid provider's market. As a consequence, users do not need to speculate on the future price of storage at all, they can just trust the oracle to discover the rent price of rent: they can provide the compensation for their quota as a "hot wallet" from which the price of rent is automatically deduced. +Let `s` and `e` be the start and end data of storage, period is `e-s` in units used in the oracle price. Let us propose the following scenario: owner `A` buys a postage batch `B` for `2^C` chunks and escrows amount `D` onto `B`. Now if we assume that price of storage per chunk per time unit at `s` is `p_s`, then we can guess that the expiry is `e'<=e` iff `D/(2^C*p_s)` then A assumes price will decrease. However, if the average price increases, then the effective period ends d units sooner if d is the smallest number such that the average price over the period [s, e] the cumulative rent goes above D `\Sum_{i=s}^e p_i*2^C >= D.` -However, this also means that undermonetising the batch in the context of increasing rent price may lead to earlier expiry. This can only be counteracted by having more funds on the batch, so users effectively pay for security (upload and disappear experience). After the intended expiry passes, all funds remaining on the batch can be used for more storage service.[^2] +In other words, increasing rent price may lead to earlier expiry. This can only be counteracted by having more funds on the batch, so users effectively pay for security (*upload and disappear experience*). After the intended expiry passes, all funds remaining on the batch can be used for more storage service.[^2] [^2]: Using the current balance to create another batch (potenitally different size, period and owner) is currently not supported but should be. In what follows we argue that instead of having fixed-quota-for-fixed-period deals as a primitive, Swarm will provide these as a solution through further smart contracts on top of the postage batches. -Surely, when users commit to a storage term, they usually have a use in mind, they want to commit to storing a particular (set of) file(s) for a period. While they understand that quota are quantised in an arbitrary way and might not match their immediate need, the surplus will often be considered waste. If users had a deal for a price for an amount for a period, leftover could be viewed as an opportunity to use the prenegotiated deal in the future. However, without such a deal, users have a hard time interpreting leftover quota on the one hand, and they expect immediate availability: a liquid market for both quota and duration. It is therefore crucial that storage providers always have available at any point, any amount for any period.[^1] +##### Availability, liquid market for storage + +When users commit to a storage term, they usually have a use in mind, they want to commit to storing a particular (set of) file(s) for a period. While they understand that quota are quantised in an arbitrary way and might not match their immediate need, the surplus will often be considered waste. If users had a deal for a price for an amount for a period, leftover could be viewed as an opportunity to use the prenegotiated deal in the future. However, without such a deal, users have a hard time interpreting leftover quota on the one hand, and they expect immediate availability: a liquid market for both quota and duration. It is therefore crucial that storage providers always have available at any point, any amount for any period.[^1] [^1]: Instead of a continuous scale of options, period is guaranteed with an accuracy of matching expected to decrease exponentially with time. -To sum up, upload and disappear experience with certainty can actually be guaranteed to users if storage providers themselves secure the availability of the chunks using their own postage batch. A liquid market is best provided if it is provided by a decentralised service network: the very swarm nodes themselves. However, it is best that the insurance or availability provider is not in the same neighbourhood as the chunk would normally be stored in yet the distribution is uniform. We argue that the insurance address of the chunk is the legacy Keccak hash of the chunk address H(addr). If the insurance address of the chunk falls exactly into the same neighbourhood as the address (well, the first byte match), then we fall back choosing the bitflip (bitwise XOR with the binary for $0xff$) on the first byte. This will place roughly one in every 256 chunk into the opposite side of the network which still keeps the balance of storage. +Upload and disappear experience with certainty can actually be guaranteed to users if storage providers themselves secure quota using their own postage batches. A liquid quota market is best provided by a *decentralised service network*: served by swarm nodes themselves. + +#### Stake providers in the address space + +The insurance or availability provider had better not be in the same neighbourhood as the chunk would normally be stored in yet their distribution should be uniform. We argue that the insurance address of the chunk is the legacy Keccak hash of the chunk address H(addr). If the insurance address of the chunk falls exactly into the same neighbourhood as the address (well, the first byte match), then we fall back choosing the bitflip (bitwise XOR with the binary for $0xff$) on the first byte. This will place roughly one in every 256 chunk into the opposite side of the network which still keeps the balance of storage. So when nodes swear on to provide insurance/availability level X service, they register their address with a stake on a (the?) staking contract. It is important that providers should be distributed evenly in neighbourhoods, ideally of the same size as that disctated by the reserve. Maybe all full nodes are providers by default. This is very useful because @@ -50,16 +95,18 @@ Some insurers may just take on the task of keeping the chunks retrieveable. For They can keep the chunk in this batch retrieveable by regularly topping up the batch. Since the hashes are uniformly distributed, this batch is expected to be naturally balanced as it should be for the node to be able to realistically fill it. The request will reference a specified expiry and the batch need to have a balance at least until that date. -The due to the increasing price of BZZ or organic decrease of the storage price, the value may increase potentially beyond what is deduced with the rent every game. This eventuality could quietly implement Arweave's promise ("pay once, store forever"), except that you can always stop it and use it to store something else. +Due either to the increasing price of BZZ or to the organic decrease of the cost of storage, the value may increase potentially beyond what is deduced with the rent every game. This eventuality could quietly implement Arweave's promise ("pay once, store forever"), except that you can always stop it and use it to store something else. Surely, the difference is that this is neither promised to happen with any amount in advance nor is the construction enforced. -These livekeepers collect and then publish chunks with their own stamp. The batch that they use will be grouping chunks by +#### Upload to the reserve: + +Insurers can collect and then publish chunks with their own stamp. The batch that they use will be grouping chunks by 1. H(address); locality of their node address 2. expiry; past request date 3. arrival time of request; As long as 3.~is undifferentiating due to a large number of (near) simultaneous requests 1 and 2 will not easily help align identity to chunk or even chunk to chunk. -This is why publishing can be delayed exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks uploaded. Problem is that if the traffic is not great, especially long term storage (more expensive), publishing may need to wait beyond the chunks of the file already disappear from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently. Surely the bigger the batch, the bigger the anonimity set, thus the better the anonimity that you get. +This is why push-syncing all the chunks in the batch can be delayed: exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks are uploaded. The problem is that if the traffic is not great, especially long term storage (more expensive), publishing may need to wait beyond the chunks of the file already disappear from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently: the bigger the batch, the bigger the anonimity set, thus, the better the anonimity that you get. In fact this opens up the possibility that livekeepers keep alive batches practically for short time periods representing a unit the others are multiples of, then they can always preserve information about how long the relevant chunks are meant to be stored. @@ -72,13 +119,12 @@ Since association by postage batch was always the wierd anomaly, solving it is a #### Livekeeper-insurer: -Overall using such a deal would allow over -gained: -even though treating the postage batch as a hot wallet has quite a few advantages (live accouting of +Even though treating the postage batch as a hot wallet has quite a few advantages, such an insurance scheme bring improvement in several areas: + - *privacy*: - - effective *stamp mixing + - effective *stamp mixing* - complete obfuscation (and plausible deniability) of chunk originator - - no leakage of shared ownership through signature by same batch owner + - no leakage of shared ownership through signature by the same batch owner - *precision*: - exact period - exacy amount @@ -101,6 +147,11 @@ even though treating the postage batch as a hot wallet has quite a few advantage + + + + + ## Specification ## Rationale From 7cd21a181e09660a1e77661c7c1c5e2b7a9cba78 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Wed, 12 Mar 2025 11:19:56 +0100 Subject: [PATCH 4/6] Update swip-35.md more elab --- SWIPs/swip-35.md | 110 +++++++++++++++++++++++++---------------------- 1 file changed, 59 insertions(+), 51 deletions(-) diff --git a/SWIPs/swip-35.md b/SWIPs/swip-35.md index 19c4303..17513fc 100644 --- a/SWIPs/swip-35.md +++ b/SWIPs/swip-35.md @@ -21,11 +21,9 @@ The current storage incentive system has a few shortcomings mainly related to th These issues are coupled with known privacy concerns regarding the shared ownership of chunks that are transparent to 3rd parties. Effectively information about the publisher as well as the chunks constituting the publication leak the system, somewhat overshadowing the otherwise rather strong privacy claims in swarm. -The latter issue is solved if the stamps on individual chunks are signed by other nodes. We will introduce *stamp mixer service* of sorts in a way that the system will not lose its capacity to prevent doubly sold postage slots.[^3] +The latter issue is solved if the stamps on individual chunks are signed by other nodes. We will introduce *stamp mixer service* of sorts in a way that the system will not lose its capacity to prevent doubly sold postage slots. -[^3]: - -On top of all this: if batches are used one per upload session etc, bookkeeping of "what chunk to link with what batch index" is less relevant; in fact, it matters solely in order to save money in case the file is reuploaded.[^4] +On top of all this: if batches are used one per upload session etc, bookkeeping of "what chunk to link with what batch index" is less relevant; in fact, it matters solely in order to save money in case the file is reuploaded. In this SWIP we introduce a scheme in which uploads are not necessarily using a stamp; even though uploaders can purchase storage for arbitrary long periods. No per-chunk metadata needs recording and made available in this case, meaning that the end-user case no longer raises portability concerns. @@ -44,6 +42,8 @@ Just like before, the swap protocol offers free (throttled) as well as paid (max ### Stamp mixing +In this section we introduce a construction we call stamp mixer, whereby users delegate the stamping of uploaded chunks to nodes in the neighbourhood designated by the hash of the chunk address. + #### Incentive to prevent double signing Postage batches primarily serve to enforce the uniform distribution of chunks, and thereby achieve a balanced load of storage across the network. As part of the solution, a user must not double spend, i.e., assign more than one chunk to the same postage slot. Originally, this is guaranteed by the fact that the owner is interested in not being caught since that may lead to the loss of their own data, i.e., data that they themselves had uploaded. If a stamp mixer of sorts is introduced, we must make sure that issuers cannot use the same slots: which happens to be the case if all chunks are insured with *negative incentives* (punitive measures). @@ -62,106 +62,114 @@ The quota encoded in the batch's slots is not neutral: as per the previous secti Incomplete utilisation can be counteracted with making sure you can rearrange between neighbourhoods by either [mining]() or having availability guaranteed by a constant influx of chunks (see section below). On the other hand, with the variance limited, the actual discount can also be made explicit in the current system and therefore [user expectations of their quota can bee managed](https://www.overleaf.com/4919957411cgrncysjqrmv#3b42ca). -#### Storage period and expiry +##### Storage period and expiry On the other hand, the redistribution game is a *probabilistic outpayment scheme* in the sense that instead of paying -- on each game all the contributors for a recurring fixed proportion of used storage we just pay out the entire reward pot for the period to a random winner at every game. This allows for central pricing sensitive to supply and offers automatic and optimal price discovery as long as we assume a liquid market of operators providing storage. As a consequence, users do not need to speculate on the future price of storage at all, they can just trust the oracle to discover the best price of rent. Users provide the compensation for their quota as a "hot wallet" from which the price of rent is automatically deduced. The postage batch is such a construct: it serves as a hot wallet. The expiry is defined as the onset of insolvency, i.e., whenever the balance goes to zero. -Let `s` and `e` be the start and end data of storage, period is `e-s` in units used in the oracle price. Let us propose the following scenario: owner `A` buys a postage batch `B` for `2^C` chunks and escrows amount `D` onto `B`. Now if we assume that price of storage per chunk per time unit at `s` is `p_s`, then we can guess that the expiry is `e'<=e` iff `D/(2^C*p_s)` then A assumes price will decrease. However, if the average price increases, then the effective period ends d units sooner if d is the smallest number such that the average price over the period [s, e] the cumulative rent goes above D `\Sum_{i=s}^e p_i*2^C >= D.` +Let `s` and `e` be the start and end data of storage, period is `e-s` in units used in the oracle price. Let us propose the following scenario: owner `A` buys a postage batch `B` for `2^C` chunks and escrows amount `D` onto `B`. Now if we assume that price of storage per chunk per time unit at `s` is `p_s`, then we can guess that the expiry is `e'<=e` iff `D/(2^C*p_s)` then A assumes price will decrease. However, if the average price increases, then the effective period ends d units sooner if d is the smallest number such that the average price over the period `[s, e]` the cumulative rent goes above D `\Sum_{i=s}^e p_i*2^C >= D.` In other words, increasing rent price may lead to earlier expiry. This can only be counteracted by having more funds on the batch, so users effectively pay for security (*upload and disappear experience*). After the intended expiry passes, all funds remaining on the batch can be used for more storage service.[^2] [^2]: Using the current balance to create another batch (potenitally different size, period and owner) is currently not supported but should be. +Due either to the increasing price of BZZ or to the organic decrease of the cost of storage, the value held on a batch may increase potentially beyond what is deduced with the rent at every game. This eventuality could quietly implement **Arweave's promise** *("pay once, store forever")*. +The differences are: + +1. the construct is not promised to happen with any amount +2. just one of the construction offered. +3. happens even if not planned +4. one can always create a new batch and use it to store something else. + + + In what follows we argue that instead of having fixed-quota-for-fixed-period deals as a primitive, Swarm will provide these as a solution through further smart contracts on top of the postage batches. -##### Availability, liquid market for storage +#### Availability, liquid market for storage quota -When users commit to a storage term, they usually have a use in mind, they want to commit to storing a particular (set of) file(s) for a period. While they understand that quota are quantised in an arbitrary way and might not match their immediate need, the surplus will often be considered waste. If users had a deal for a price for an amount for a period, leftover could be viewed as an opportunity to use the prenegotiated deal in the future. However, without such a deal, users have a hard time interpreting leftover quota on the one hand, and they expect immediate availability: a liquid market for both quota and duration. It is therefore crucial that storage providers always have available at any point, any amount for any period.[^1] +When users commit to a storage term, they usually have a use in mind, they want to commit to storing a particular (set of) file(s) for a particular period (importance). While they understand that quota are quantised in an arbitrary way and might not match their immediate need, the surplus will often be considered waste. If users had a deal for a price for an amount for a period, leftover could be viewed as an opportunity to use the prenegotiated deal in the future. However, without such a deal, users have a hard time interpreting leftover quota on the one hand, and they expect immediate availability: a liquid market for both quota and duration. It is therefore crucial that storage providers always have available at any point, any amount for any period.[^1] [^1]: Instead of a continuous scale of options, period is guaranteed with an accuracy of matching expected to decrease exponentially with time. -Upload and disappear experience with certainty can actually be guaranteed to users if storage providers themselves secure quota using their own postage batches. A liquid quota market is best provided by a *decentralised service network*: served by swarm nodes themselves. +Upload and disappear experience with certainty can actually be guaranteed to users if storage providers themselves secure quota using their own postage batches. + +This construction is virtually a stamp mixer in that it blinds any originator address's association with chunks and also obfuscates the belonging together of a set of chunks. +Since association by postage batch was always the wierd anomaly, solving it is a major improvement in swarm's privacy. -#### Stake providers in the address space -The insurance or availability provider had better not be in the same neighbourhood as the chunk would normally be stored in yet their distribution should be uniform. We argue that the insurance address of the chunk is the legacy Keccak hash of the chunk address H(addr). If the insurance address of the chunk falls exactly into the same neighbourhood as the address (well, the first byte match), then we fall back choosing the bitflip (bitwise XOR with the binary for $0xff$) on the first byte. This will place roughly one in every 256 chunk into the opposite side of the network which still keeps the balance of storage. +#### Staked providers in the address space + +A liquid quota market is best provided by a *decentralised service network*: served by swarm nodes themselves. +The insurance or availability provider had better not be in the same neighbourhood as the chunk would normally be stored and yet, their distribution should be uniform. We propose that the insurance address of the chunk is the legacy Keccak hash of the chunk address `H(addr)`. If the insurance address of the chunk falls exactly into the same neighbourhood as the address (well, the first byte match), then we fall back choosing the bitflip (bitwise XOR with the binary for $0xff$) on the first byte. This will place roughly one in every 256 chunk into the opposite side of the network which still keeps the balance of storage. + +So when nodes swear on to provide insurance (or insurance/availability level X service, they register their address with a stake on a (the?) staking contract. It is important that providers should be distributed evenly in neighbourhoods, ideally of the same size as that dictated by the reserve. +Maybe all full nodes are providers by default. This is very useful because when a node provides the service, they expect chunks to their neighbourhood by the hash of the BMT address of a CAC --- from now a *new chunk type*. Once they receive it, they must provide the uploader with a signed receipt as an acknowledgment which they send backwarding on the same route. -So when nodes swear on to provide insurance/availability level X service, they register their address with a stake on a (the?) staking contract. It is important that providers should be distributed evenly in neighbourhoods, ideally of the same size as that disctated by the reserve. -Maybe all full nodes are providers by default. This is very useful because -when a node provides the service, they expect chunks to their neighbourhood by the hash of the BMT address of a CAC --- from now a new chunk type. Once they receive it, they -must provide the uploader with a signed receipt. Some insurers may just take on the task of keeping the chunks retrieveable. For this, they need to create a postage batch matching the period requirement in the chunk request and eventually sign stamps and send the chunk with the stamp to their destination. -They can keep the chunk in this batch retrieveable by regularly topping up the batch. Since the hashes are uniformly distributed, this batch is expected to be naturally balanced as it should be for the node to be able to realistically fill it. -The request will reference a specified expiry and the batch need to have a balance at least until that date. +They can keep the chunk in this batch retrieveable by regularly topping up the batch if needed. Since the hashes are uniformly distributed, this batch is expected to be naturally balanced as it should be. So for the node, it is quite realistically to be able to fill it. +The request will reference a specified expiry and the batch needs to have a balance at least until that date. -Due either to the increasing price of BZZ or to the organic decrease of the cost of storage, the value may increase potentially beyond what is deduced with the rent every game. This eventuality could quietly implement Arweave's promise ("pay once, store forever"), except that you can always stop it and use it to store something else. Surely, the difference is that this is neither promised to happen with any amount in advance nor is the construction enforced. +#### Upload to the reserve, filling batches -#### Upload to the reserve: +Insurers can collect and then publish chunks with their own stamp. The batch that they use will be grouping chunks by -Insurers can collect and then publish chunks with their own stamp. The batch that they use will be grouping chunks by 1. H(address); locality of their node address -2. expiry; past request date -3. arrival time of request; +2. end; datetime of expiry of storage +3. start; arrival time of request -As long as 3.~is undifferentiating due to a large number of (near) simultaneous requests -1 and 2 will not easily help align identity to chunk or even chunk to chunk. -This is why push-syncing all the chunks in the batch can be delayed: exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks are uploaded. The problem is that if the traffic is not great, especially long term storage (more expensive), publishing may need to wait beyond the chunks of the file already disappear from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently: the bigger the batch, the bigger the anonimity set, thus, the better the anonimity that you get. +As long as 3. is undifferentiating due to a large number of (near) simultaneous requests +1 and 2 will not help align identity to chunk or even chunk to chunk easily. This is why push-syncing all the chunks in the batch can be delayed: exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally, the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks are uploaded. The problem is that if the traffic is not great, especially long term storage (more expensive), publishing may need to wait beyond the chunks of the file already disappear from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently: the bigger the batch, the bigger the anonimity set, thus, the better the anonimity that you get. -In fact this opens up the possibility that livekeepers keep alive batches practically for short time periods representing a unit the others are multiples of, then they can always preserve information about how long the relevant chunks are meant to be stored. - -This construction is virtually a stamp mixer in that it blinds any originator address's association with chunks and also obfuscates the belonging together of a set of chunks. -Since association by postage batch was always the wierd anomaly, solving it is a major improvement in swarm's privacy. +In fact, this opens up the possibility that providers keep alive batches practically for short time periods representing a unit period that the others are multiples of. In order to know how long the relevant chunks are meant to be stored, +nodes save information about the original request. -#### Insurer: +## Specification -#### Livekeeper-insurer: +## Evaluation Even though treating the postage batch as a hot wallet has quite a few advantages, such an insurance scheme bring improvement in several areas: - - *privacy*: + - **privacy**: - effective *stamp mixing* - complete obfuscation (and plausible deniability) of chunk originator - no leakage of shared ownership through signature by the same batch owner - - *precision*: + - **precision**: - exact period - exacy amount - - *predictability*: - - easier to understand pprimitives - - *preparedness*: + - **predictability**: + - easier to understand primitives + - **preparedness**: - real *upload and disappear* experience (no need to be around to secure) - immediate availability of uploads for any amount/period without wait or prepurchase - - *pricing*: - - easier to understand/more pricing + - **pricing**: + - free upload into cache + - easier to understand pricing - potentially complete utilisation of batches leading to cheaper prices or easier to coordinate subsidies - prepurchase (keeping batches open) makes sense (speculation on price deal) - moderate future price speculation needed - 'pay once, store forever' now available to providers (only) - no benefit from rent price drop - - *portability*: + - **portability**: - only providers need to keep track of batches and batch utilisation - since users got all they need from just knowing that a file/root was insured/kept alive +## Implementation notes and roadmap +Due to the complexity of this epic, components will be discussed in separate SWIPs: +- free postage +- global pinning, incentives, notifications +- stamp mixing, requests, receipts and address indexes +- challanges with inclusion proofs etc +The implementation notes, test cases, tooling support will be detailed in the individual SWIPs for the components. +### Backward compatibility +Due to the new chunktype we define, implementing this swip will introduce **backward incompatibility** in the code. - - -## Specification - -## Rationale - -## Backward compatibility - -## Test Cases - -## Implementation - ## Copyright From d36c674911757a46c02bb45db70782ad282fbe5e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Tue, 22 Apr 2025 16:19:05 +0200 Subject: [PATCH 5/6] Update swip-35.md --- SWIPs/swip-35.md | 277 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 213 insertions(+), 64 deletions(-) diff --git a/SWIPs/swip-35.md b/SWIPs/swip-35.md index 17513fc..a40a615 100644 --- a/SWIPs/swip-35.md +++ b/SWIPs/swip-35.md @@ -10,136 +10,286 @@ created: 2025-03-09 --- # Advanced storage guarantees - + + ## Abstract -This SWIP proposes an entire epic about adding further layers of incentives to the basic positive incentives in Swarm. By adopting the SWIP, swarm will offer a wider range of constructions for its users to secure the persistance of chunks and their metadata and at the same time allow a low barrier of entry service that node operators with extra storage or bzz future providers are able to increase their revenue with. +This SWIP proposes an entire epic about adding further layers of incentives to the basic positive incentives in Swarm. By adopting this SWIP, swarm will offer a wider range of constructions for its users to secure the persistance of chunks and their metadata and at the same time allow a low-barrier-of-entry service for node operators.[^55] + +[^55]: Allowing for a broader range of hardware specs that operators can run their client software on. ## Motivation -The current storage incentive system has a few shortcomings mainly related to the problem that postage batches are difficult to interpret. This interpretation problem leads to quite a bit of misunderstanding and misguided (also potentially failing) expectations regarding both the stored amount and the storage period. +The current storage incentive system has a few shortcomings mainly related to the following areas: +1. **parameters of on-chain stamp purchase** -- +Users typically have a hard time interpreting the transaction parameters required when buying a batch of postage stamps on the blockchain. Users need a clear interpretation of + - insured amount + - insured period + - mutability +1. **guarantees of durability and retrievability** -- +Users seek a wider range of options regarding the security guarentees for their uploaded content. Users and operators must be given adequate tooling for + - upload to cache only + - storage insurance + - community-maintained data (global pinning) + - cheap cold storage +2. **privacy** -- +Postage stamps self-signed by the uploader as well as using a single batch per session leak information which weaken plausible deniability and open the door for targetted withholding or ransome attacks. Stamps leak + - identity of uploader + - co-constituency of chunks (which chunks belong together in a file, directory or upload session) + +#### Interpreting postage batch parameters + +Lack of clarity regarding the interpretation of batch parameters has led to misguided expectations about the expiry of persistance guarantees as well as about the available amount insured. Mutability is also confusing users when it came to the reuse of postage slots. + +#### Privacy + +The signature on a stamp serves as *proof of relevance*[^2] when it comes to evidence submitted as part of the claim transaction. As long as stamps are self-signed by the uploader, the batch owner can be identified as the originator of the data for 3rd parties. + +[^2]: The validation of a batch owner associating a chunk with a postage batch serves to proof that someone is indeed willing to pay for the data in question. -These issues are coupled with known privacy concerns regarding the shared ownership of chunks that are transparent to 3rd parties. Effectively information about the publisher as well as the chunks constituting the publication leak the system, somewhat overshadowing the otherwise rather strong privacy claims in swarm. +Somewhat independently of this, the fact that same-owner batches are used when uploading a file, the chunks uploaded during one session leak co-constituency, i.e., which chunks are likely to constitute one file, directory or belong together in a single upload session. This overshadows somewhat Swarm's otherwise rather strong privacy claims. -The latter issue is solved if the stamps on individual chunks are signed by other nodes. We will introduce *stamp mixer service* of sorts in a way that the system will not lose its capacity to prevent doubly sold postage slots. -On top of all this: if batches are used one per upload session etc, bookkeeping of "what chunk to link with what batch index" is less relevant; in fact, it matters solely in order to save money in case the file is reuploaded. -In this SWIP we introduce a scheme in which uploads are not necessarily using a stamp; even though uploaders can purchase storage for arbitrary long periods. -No per-chunk metadata needs recording and made available in this case, meaning that the end-user case no longer raises portability concerns. +#### Insurance -Last, but not least, enforcing payments for each upload does not really make sense: firstly, users had some difficulty to grasp that given the free-tier for bandwidth inentives allowing free downlad and upload with throttled throughput, how come they need to pay for the storage. This problem was most obvious for really short-lived messages that are used in chat or notifications. Since, in these cases, preservation is not meaningful or even actively not desired, paying for postage sounds not only a unnecessary but outright wasteful. Secondly, the fact that expired chunks are not deleted but are put to the *cache*[^5] poses the question how come there is no direct entry to it. -This anomaly can be solved if we introduce free uploads, i.e., uploading chunks to Swarm without a postage stamp. +The storage incentives currently present in Swarm use an on-chain game which orchestrates the fair redistribution of postage payments to storers, thereby providing *positive incentivisation* at a collective level. Such a system, however, is suspect to *the tragedy of the commons* problem in that disappearing content will have no negative consequence to storers. Any particular chunk missing currently does not impose any punitive measures on the one or several storers who explicitly promised to store the chunk.[^6] +[^6]: Alternatively, such an insurance system can be provided as an insurance market place where the availability must be guaranteed through dynamically adopted to??? -[^5]: so that popular content can bring bandwidth revenue even if no longer available for storage inecntive rewards. +#### Global pinning, + +It is up to the insurer node to decide how they guarantee the integrity of the entire set of chunks available from any particular root hash reference. They got a few options where they keep the contents: +- keep the content locally pinned under the root +- keep the content stored on swarm.[^15] +- use other external services like Filecoin, Arweave or Storj. + +[^15]: In this case, the content may be reencrypted and protected with erasure codes and data avaiability sampling + +Swarm would benefit from the availability of such options either with or without retrievability guarantees (*warm* vs *cold* storage). + +#### Free uplods to cache + +Messages of real-time chat or notifications or frequently updated indexes without history are just obvious examples of data that are meant to be short-lived. Since, in these cases, preservation beyond temporary caching is unnecessary, paying for postage is wasteful. +Such content meant not to be stored should just be be uploadable without a postage stamp and put to the cache and not to the reserve. In order to mitigate targetted attacks, bandwidth compensation must be adapted to express as well as transfer the cost between uploader and storer in case postage stamps are not able to. ## Solution -### Free uploads to cache +### Negative incentives +The lack of individual accountability makes Swarm's storage incentivisation limited as a security measure against data loss. Introducing non-compensatory insurance, on the other hand, adds an additional layer of *negative incentives*. In other words, the threat of punitive measures (losing the stake deposited against foul play) is introduced +in order to compell storage providers to ensure reliability of data retention and/or prompt retrievability for users. + +### Stampless uploads to cache -Just like before, the swap protocol offers free (throttled) as well as paid (max throughput) service through cheques, now, upload also offers free (chunks are put to cache and purged if necessary) as well as paid (chunk put to the reserve until stamp expires) service using postage stamps. +When expired chunks are removed from a node's reserve, they are not deleted but rather, are put to the *cache*[^5]. This suggests that chunks uploaded without a stamp should also start there (see [SWIP-36](https://github.com/ethersphere/SWIPs/pull/70)). + +[^5]: so that popular content can bring bandwidth revenue even if no longer available for storage incentive rewards. + + ### Stamp mixing -In this section we introduce a construction we call stamp mixer, whereby users delegate the stamping of uploaded chunks to nodes in the neighbourhood designated by the hash of the chunk address. +Privacy issues related to signing stamps can be solved by introducing a construction we call *$tumbler (=stamp tumbler)* which, similarly to a currency *mixer*, stamp the chunks delegated to them by the uploader of the chunks, thereby obfuscating their identity. + +This delegation is most efficient if obfuscation is done +in the neighbourhood designated by the *secondary chunk address* that is different from the primary address which in turn designates the neighbourhood that the chunk is retrievable from. -#### Incentive to prevent double signing +## Rationale -Postage batches primarily serve to enforce the uniform distribution of chunks, and thereby achieve a balanced load of storage across the network. As part of the solution, a user must not double spend, i.e., assign more than one chunk to the same postage slot. Originally, this is guaranteed by the fact that the owner is interested in not being caught since that may lead to the loss of their own data, i.e., data that they themselves had uploaded. If a stamp mixer of sorts is introduced, we must make sure that issuers cannot use the same slots: which happens to be the case if all chunks are insured with *negative incentives* (punitive measures). #### Contractual terms of storage -The most natural way to pay for storage is by the *amount* and *time period*. Surely, the unit price of storage (per chunk, per block) already suggests this. However, for most, a deal (through an act of purchase) entails that the counterparty service can be kept to the terms *precisely*: i.e., you bought a fixed quota (amount of bytes) for a fixed period (duration starting now, expiring after a fixed period or on a fixed datte). Experience shows that any deviation from this expectation comes as a surprise for the user. +The most natural way to pay for storage is to interpret it as *fixed period capacity rent* and expect parameters to be the *amount of capacity* and the *time period* (5Gb from no until a year from now) as the unit price of storage (per chunk, per block) already suggests. However, for most, a deal (through an act of batch purchase) entails that the counterparty service can be kept to the terms *precisely*: i.e., you bought a fixed *quota* (amount of bytes, 5 million) for a fixed period (duration starting now, expiring after a fixed period of a year). Experience shows that any deviation from this expectation comes as a surprise for the user. + + + +##### Mutability and Incentive to prevent double signing + +Postage batches primarily serve to enforce the uniform distribution of chunk addresses, and thereby achieve a balanced load of storage across the network. This is solved by partitioning the `2^b` storage slots of a batch (of depth `b`) into `2^u` equal-sized buckets (of depth `2^{b-u}`). The stamp contains the index of the slot composed of a bucket index and a within-bucket index. Batches also impose a constraint on the stamps' validity: the owner must not assign more than one chunk to the same postage slot.[^11] + +[^11]: Meaning that doing so will invalidate an immutable batch, make retrievals potentially fail in case the single chunk is not well-defined. + +Double signing of a slot (though detectable[^17]) will only be avoided if the owner is incentivised to avoid it. Originally, since uploaders are themselves the owners of the batch they use for stamping, they are naturally incentivised to avoid double signing which could threaten retrievability of the content that they upload. If stamping is delegated to a stamp tumbler, double signing must be avoided through negative incentives (the threat of losing their stake). + + +[^17]: The number of buckets must at all times be greater than the number of neighbourhoods, so each bucket belong in a neighbourhood, i.e., all the chunks assigned to a slot in a particular bucket are stored within the neighbourhood (as the nodes' area of responsibility), which guarantees that the double signing of a slot is detectable by any node in the neighbourhood. -The current primitive of storage compensation, the postage batch falls short on these. The primary reason is that it was designed with the simplicity in mind pertaining to the intergration into the redistribution system. ##### Postage batch utilisation -The quota encoded in the batch's slots is not neutral: as per the previous section, it also serves to force balanced use across neighbourhoods: maxing out the subquota for one neighbourhood implies maxing out the same subquota for each neighbourhood which requires no variance across NH-s; so the effective batch utilisation rate increases with overall volume, in other words there is a natural overhead for small batches[^1] +The quota represented by the set of storage slots of a batch is evenly distributed in a number of buckets, as it also serves to force balanced use across neighbourhoods. +Even though the most effecient utilisation of a batch is if stamps can max out each bucket, already a single bucket filled up will make an immutable batch practically not usable since, as it has no slots available in the bucket filled, it cannot be guaranteed to stamp just any chunk.[^31] + +[^31] Incomplete utilisation can be counteracted with making sure you can reassign the +'same' chunk to another bucket by either [mining]() or having availability guaranteed by a constant influx of chunks (see section below). On the other hand, with the variance limited, the actual discount can also be made explicit in the current system, and, therefore [user expectations of their quota purchased that is suggested bt the batch depth must be modulated by factoring in the packed address chunks needed to represent files, and the manifest nodes to serve as directories, erasure code parities + +Instead the volume called [*effective utilisation*](https://www.overleaf.com/4919957411cgrncysjqrmv#3b42ca) must be given to users. +The effective batch utilisation rate increases with overall volume implying a natural overhead for small batches[^1] -[^1]: or more naturally put, there is wholesale bulk discount. +[^1]: or more naturally put, there is an incerasing discount rate when purchasing larger volumes. -Incomplete utilisation can be counteracted with making sure you can rearrange between neighbourhoods by either [mining]() or having availability guaranteed by a constant influx of chunks (see section below). On the other hand, with the variance limited, the actual discount can also be made explicit in the current system and therefore [user expectations of their quota can bee managed](https://www.overleaf.com/4919957411cgrncysjqrmv#3b42ca). ##### Storage period and expiry On the other hand, the redistribution game is a *probabilistic outpayment scheme* in the sense that instead of paying -- on each game all the contributors for a recurring fixed proportion of used storage we just pay out the entire reward pot for the period to a random winner at every game. This allows for central pricing sensitive to supply and offers automatic and optimal price discovery as long as we assume a liquid market of operators providing storage. As a consequence, users do not need to speculate on the future price of storage at all, they can just trust the oracle to discover the best price of rent. Users provide the compensation for their quota as a "hot wallet" from which the price of rent is automatically deduced. The postage batch is such a construct: it serves as a hot wallet. The expiry is defined as the onset of insolvency, i.e., whenever the balance goes to zero. -Let `s` and `e` be the start and end data of storage, period is `e-s` in units used in the oracle price. Let us propose the following scenario: owner `A` buys a postage batch `B` for `2^C` chunks and escrows amount `D` onto `B`. Now if we assume that price of storage per chunk per time unit at `s` is `p_s`, then we can guess that the expiry is `e'<=e` iff `D/(2^C*p_s)` then A assumes price will decrease. However, if the average price increases, then the effective period ends d units sooner if d is the smallest number such that the average price over the period `[s, e]` the cumulative rent goes above D `\Sum_{i=s}^e p_i*2^C >= D.` +Let `s` and `e` be the start and end date of storage, period is `e-s` in units used in the oracle price. Let us propose the following scenario: owner `A` buys a postage batch `B` for `2^C` chunks and escrows amount `D` onto `B`. Now if we assume that price of storage per chunk per time unit at `s` is `p_s`, then we can guess that the expiry is `e'<=e` iff `D/(2^C*p_s)` then A assumes price will decrease. However, if the average price increases, then the effective period ends d units sooner; if d is the smallest number such that the average price over the period `[s, e]` the cumulative rent goes above D `\Sum_{i=s}^e p_i*2^C >= D.` -In other words, increasing rent price may lead to earlier expiry. This can only be counteracted by having more funds on the batch, so users effectively pay for security (*upload and disappear experience*). After the intended expiry passes, all funds remaining on the batch can be used for more storage service.[^2] +In other words, increasing rent price may lead to earlier expiry. This can only be counteracted by having more funds on the batch, so users effectively pay for security (*upload and disappear experience*). If the intended expiry passes without the batch running out of funds, then storage just continues, or alternatively all funds remaining on the batch can be used for more storage service.[^12] -[^2]: Using the current balance to create another batch (potenitally different size, period and owner) is currently not supported but should be. +[^12]: Using the current balance to create another batch (potenitally different size, period and owner) is currently not supported but it should be. -Due either to the increasing price of BZZ or to the organic decrease of the cost of storage, the value held on a batch may increase potentially beyond what is deduced with the rent at every game. This eventuality could quietly implement **Arweave's promise** *("pay once, store forever")*. +Due either to the increasing price of BZZ or to the organic decrease of the cost of storage, the value held on a batch may increase potentially at a rate beyond what is countered by rent deduced at every game. This eventuality could quietly implement **Arweave's promise** *("pay once, store forever")*. The differences are: -1. the construct is not promised to happen with any amount -2. just one of the construction offered. -3. happens even if not planned -4. one can always create a new batch and use it to store something else. +1. the construct is not promised to happen with just any amount, +2. with the right amount it happens even if not planned, +3. it is just one of the construction offered, +4. the funds on a batch can be used to seed fund a new batch and store other things In what follows we argue that instead of having fixed-quota-for-fixed-period deals as a primitive, Swarm will provide these as a solution through further smart contracts on top of the postage batches. -#### Availability, liquid market for storage quota +#### Availability of storage quota -When users commit to a storage term, they usually have a use in mind, they want to commit to storing a particular (set of) file(s) for a particular period (importance). While they understand that quota are quantised in an arbitrary way and might not match their immediate need, the surplus will often be considered waste. If users had a deal for a price for an amount for a period, leftover could be viewed as an opportunity to use the prenegotiated deal in the future. However, without such a deal, users have a hard time interpreting leftover quota on the one hand, and they expect immediate availability: a liquid market for both quota and duration. It is therefore crucial that storage providers always have available at any point, any amount for any period.[^1] +When users commit to a storage term, they usually have a use in mind, they want to commit to storing a particular (set of) file(s) for a particular period (importance). While they understand that quota are quantised in a way that might not match their immediate need, and will leave them with leftover storage slots, a surplus.[^22] +On the other hand, uploaders prefer immediate availability of any quota for any period. -[^1]: Instead of a continuous scale of options, period is guaranteed with an accuracy of matching expected to decrease exponentially with time. +[^22]: If users had a deal for a price for an amount for a period, leftover could be viewed as an opportunity to use the prenegotiated deal in the future when prices are higher. However, without such a deal, users have a hard time interpreting leftover quota for anything else than waste. -Upload and disappear experience with certainty can actually be guaranteed to users if storage providers themselves secure quota using their own postage batches. +If insurance providers themselves are to secure the quota using their own postage batch, then the surplus is all the more likely to be filled. Completely filled batches result in a lower unit prices of storage. -This construction is virtually a stamp mixer in that it blinds any originator address's association with chunks and also obfuscates the belonging together of a set of chunks. +The stumbler construction is virtually a stamp mixer in that it blinds any originator address's association with chunks and also obfuscates their co-constituency (belonging together of a set of chunks). Since association by postage batch was always the wierd anomaly, solving it is a major improvement in swarm's privacy. -#### Staked providers in the address space +#### Distribution of responsibility: providers in the overlay address space + +A liquid quota market is best provided by a *decentralised service network*: served by swarm nodes themselves. Ideally providers will have a density not too dissimilar to the nodes providing the reserve, but the system also needs to be prepared to provide service in the case of a lot (a couple of orders of magnitude) fewer or more nodes. This is taken care of by the swear contract, which maintains a balanced set of providers according to *smart neighbourhood management* ([described in a separate SWIP (SWIP-39)](https://github.com/ethersphere/SWIPs/pull/74)) -A liquid quota market is best provided by a *decentralised service network*: served by swarm nodes themselves. -The insurance or availability provider had better not be in the same neighbourhood as the chunk would normally be stored and yet, their distribution should be uniform. We propose that the insurance address of the chunk is the legacy Keccak hash of the chunk address `H(addr)`. If the insurance address of the chunk falls exactly into the same neighbourhood as the address (well, the first byte match), then we fall back choosing the bitflip (bitwise XOR with the binary for $0xff$) on the first byte. This will place roughly one in every 256 chunk into the opposite side of the network which still keeps the balance of storage. +A *withholding attack* is when an insurer is grieving[^7] consumers by only providing the chunk if challenged. If a chunk is to be retrievable from a different neighbourhood (belong to the reserve based on the chunk's address) than the one that insures it, then the onus of retrievability falls on nodes that are independent in terms of operation. -So when nodes swear on to provide insurance (or insurance/availability level X service, they register their address with a stake on a (the?) staking contract. It is important that providers should be distributed evenly in neighbourhoods, ideally of the same size as that dictated by the reserve. -Maybe all full nodes are providers by default. This is very useful because when a node provides the service, they expect chunks to their neighbourhood by the hash of the BMT address of a CAC --- from now a *new chunk type*. Once they receive it, they must provide the uploader with a signed receipt as an acknowledgment which they send backwarding on the same route. +[^7]: a grieving attack is where the adversary is able to make their victim(s) suffer losses, time or other inconvenience without them gaining. -Some insurers may just take on the task of keeping the chunks retrieveable. For this, they need to create a postage batch matching the period requirement in the chunk request and eventually sign stamps and send the chunk with the stamp to their destination. +We propose that chunks have a *secondary address*, defined as the (legacy Keccak) hash of the primary chunk address with or without the first bit flipped (bitwise XOR the first byte with 0x80). This will place roughly one in every 256 chunk into the opposite side of the network which still keeps the balance of storage. -They can keep the chunk in this batch retrieveable by regularly topping up the batch if needed. Since the hashes are uniformly distributed, this batch is expected to be naturally balanced as it should be. So for the node, it is quite realistically to be able to fill it. +So when nodes swear on to provide insurance (or insurance/availability level X service), they register their address with a stake on a (the?) staking contract. It is important that providers should be distributed evenly in neighbourhoods, ideally of the same size as that dictated by the reserve. + +Maybe all full nodes are providers by default. This is very useful because when a node provides the service, they expect chunks to their neighbourhood by the secondary address (the hash of the BMT address) of a CAC --- from now a *new chunk type*. Once they receive it, they must provide the uploader with a storage receipt as an acknowledgment which they send back to the originator using the same route (backwarding). + +Some insurers may just take on the task of keeping the chunks retrieveable. For this, they need to keep a postage batch alive that issued the stamp of the chunk, throughout the period matching the requirement in the chunk request. + +They can keep the chunk in this batch retrieveable by regularly topping up the batch if needed. Since the hashes are uniformly distributed, this batch is expected to be naturally balanced as it should be. So it is quite realistic for the node to be able to fill it. The request will reference a specified expiry and the batch needs to have a balance at least until that date. -#### Upload to the reserve, filling batches -Insurers can collect and then publish chunks with their own stamp. The batch that they use will be grouping chunks by +### Insurance + +#### Punitive measures + +Unlike in the case of bandwidth incentives where retrievals are immediately accounted and settled, long-term storage guarantees are promissory in nature and do not allow the instant gratification of forwarding/backwarding. It is only after the end of its validity that one can check if a promise has been kept. +Nodes wanting to sell storage space should have a stake verified and locked-in at the time of making their promise. This implies that nodes need to register in advance, agreeing to the contract by placing a security deposit. Once registered, a node may start selling storage promises. While their registration is active, if they are found to have lost a chunk that was covered by their promise, they stand to lose their deposit. + + +Particular attention is required in the design of the incentive system to make sure that failure to store every last bit promised is not only unprofitable but outright catastrophic to the insurer.[^16] + +[^16]: Merely losing reputation is not sufficient as a +deterrent against foul play in these instances: since new nodes must be allowed to provide services right away, cheaters could just resort to new identities and keep selling (empty) storage promises. + + +Now we will elaborate on a class of incentive schemes we call +"swap, swear, and swindle" due to the basic components: + +- *SWAP*: The swarm accounting protocol makes sure that peers exchange chunks +and receipts, triggering swap accounting and settlement. This must be used for payment +in order for true uploaders to remain anonymous. +- *SWEAR*: Storer nodes register with their stake and make an implicit promise to store the chunks whose secondary address falls within their area of responsibility. +- *SWINDLE* -- Storers can be challanged on their promise through an on-chain litigation process. + +In case an escrow is used, the reward is vested over a period of +the storer is only awarded the locked funds if they are able +to provide *proof of entitlement*. This procedure is analogous to the postage +stamp redistribution game. Out-payments conditional on proofs of custody can be implemented in a way similar to the reserve redistribution game. -1. H(address); locality of their node address -2. end; datetime of expiry of storage -3. start; arrival time of request +The SWEAR contract allows nodes to register their public key to become +accountable participants in the swarm. Registration involves sending +the deposit to the SWEAR contract, which serves as collateral in case +the terms that registered nodes "swear" to keep are violated (i.e. nodes +do not keep their promise to store). The registration is open-ended Users of Swarm should be able to count on the loss of deposit as +a disincentive against foul play for as long as the insurer node has open commitments. +Therefore, even a well-performing Swarm node is only eligible to reclaim their +deposit after all their commitments are closed or taken over by new nodes. -As long as 3. is undifferentiating due to a large number of (near) simultaneous requests -1 and 2 will not help align identity to chunk or even chunk to chunk easily. This is why push-syncing all the chunks in the batch can be delayed: exactly in order to achieve that a larger number of simultaneous requests get published together. Ideally, the time of delay coincides with the time of batch completion, i.e., whenever the batch (prepared to support the time until original uploaders want the chunk to be stored) is fully utilised, it is closed and all the chunks are uploaded. The problem is that if the traffic is not great, especially long term storage (more expensive), publishing may need to wait beyond the chunks of the file already disappear from the swarm's cache. These chunks therefore need to be first uploaded with a rather short period just to survive and later maybe also have their subscription renewed. Such a strategy creates more usage for the short periods which then leads to the ability to fill bigger batches faster and more efficiently: the bigger the batch, the bigger the anonimity set, thus, the better the anonimity that you get. +#### Challenge -In fact, this opens up the possibility that providers keep alive batches practically for short time periods representing a unit period that the others are multiples of. In order to know how long the relevant chunks are meant to be stored, -nodes save information about the original request. +If a node fails to observe the rules of the Swarm they swear to keep, they are to face +the enforcement of punitive measures. This must be preceded by an on-chain litigation +procedure called *SWINDLE (Secured With INsurance Deposit Litigation and Escrow)*. +When a user attempts to retrieve insured content and fails to find a +chunk, they can report the loss by submitting a challenge analogously to a court +case in which the insurers the defendants who are guilty until proven innocent. Similarly to a court procedure, public litigation on the blockchain should be a last resort when the rules have been abused despite the deterrents and positive incentives. +The challenge takes the form of a transaction sent to the SWINDLE +contract, in which the challenger presents evidence implicating a particular node having made a promise to insure that particular chunk for a period that has not ended yet. + +Any node is allowed to send a challenge for a chunk as long as +they have evidence available to back the claim. The validity of the challenge as well as its refutation must be easily verifiable by the contract. The contract +verifies challenge by checking the following conditions: +1. *covered* -- The chunk in question is (implicitly) covered by the insurance. +1. *authentic* -- The insurance was signed with the key the node registered with. +2. *active* -- The expiry date of the insurance has not passed. +3. *funded* -- Sufficient funds are sent in the transaction[^13] + + +[^13]: in order to compensate the insurer for uploading the chunk in case of a refuted challenge. The payment sent with the transaction is to disincentivise frivolous litigation, +i.e. bombarding the blockchain with bogus challenges and potentially +causing a DoS attack. + +Upon succcessful challenge, an event is logged by the contract to notify the accused of their obligation to refute. The challenge remains open for a fixed time period, the end of which is the deadline for refuting the challenge. Refutation itself consists of uploading the entire chunk. This (1) serves as 3rd party verifiable proof that they keep their promise as well as (2) makes the disputed data available to nodes that require it. +Upon successful refutation, the challenge is cleared from the blockchain state. While the stake of the accused node remains untouched, the cost of uploading +the chunk must be borne by the challenger and is paid from the deposit. + + +If the deadline passes without successful refutation of the challenge, +then the charge is regarded as proven and the case enters into the en- +forcement stage. Nodes that are proven guilty of losing a chunk must lose +their stake. Enforcement is guaranteed to be successful by the fact +that nodes' stakes are kept locked up (in the SWEAR contract). +If, on litigation, it turns out that a chunk (that was covered) +was lost, the deposit must be at least partly burned.[^4] + +[^4]: Note that this is necessary because, if penalties were paid out as compensation, it would provide an avenue of early exit for a registered node by "losing" bogus chunks that had been challenged on by colluding users. + +#### Incentivising promissory services +Delayed payments without locked funds leave storers vulnerable to +non-payment. Advance payments (i.e. payments settled at the time of +contracting, not after the storage period ends) on the other hand, leave +the buyers vulnerable to cheating, since after collecting more from sales than their stake, they can afford losing it and they still walk away with a profit even without doing any work. +Stake must be set high enough so this attack is no longer economical. +When paying for the insurance, the amount for the entire period must be paid in full to eliminate the insurer’s distrust due to potential insolvency. This amount is then released in installments on the condition that the node provides a proof of custody. ## Specification + + + ## Evaluation -Even though treating the postage batch as a hot wallet has quite a few advantages, such an insurance scheme bring improvement in several areas: +Even though treating the postage batch as a hot wallet has quite a few advantages, the proposed insurance scheme will bring improvement in several areas: - **privacy**: - effective *stamp mixing* - - complete obfuscation (and plausible deniability) of chunk originator - - no leakage of shared ownership through signature by the same batch owner + - true *permissionlessness* and *plausible deniability* of uploads, i.e., complete obfuscation of the chunk originator + - no leakage of co-constituency (infrmation that chunks belong to the same file) - **precision**: - - exact period - - exacy amount + - exact period of data retention + - exact amount: what you insure is what you pay for. - **predictability**: - - easier to understand primitives + - easier to understand parameters yet primitives are pure - **preparedness**: - real *upload and disappear* experience (no need to be around to secure) - immediate availability of uploads for any amount/period without wait or prepurchase @@ -149,26 +299,25 @@ Even though treating the postage batch as a hot wallet has quite a few advantage - potentially complete utilisation of batches leading to cheaper prices or easier to coordinate subsidies - prepurchase (keeping batches open) makes sense (speculation on price deal) - moderate future price speculation needed - - 'pay once, store forever' now available to providers (only) + - 'pay once, store forever' now available - no benefit from rent price drop - **portability**: - only providers need to keep track of batches and batch utilisation - - since users got all they need from just knowing that a file/root was insured/kept alive + - implicit storage receipts, ## Implementation notes and roadmap -Due to the complexity of this epic, components will be discussed in separate SWIPs: -- free postage +Due to the complexity of this epic, components will be discussed in separate SWIPs (together implementation notes, test cases, tooling support): +- stampless upload - [SWIP-36](https://github.com/ethersphere/SWIPs/pull/70) +- smart neighbourhood management - [SWIP-39](https://github.com/ethersphere/SWIPs/pull/74) - global pinning, incentives, notifications - stamp mixing, requests, receipts and address indexes -- challanges with inclusion proofs etc -The implementation notes, test cases, tooling support will be detailed in the individual SWIPs for the components. +The will be detailed in the individual SWIPs for the components. ### Backward compatibility -Due to the new chunktype we define, implementing this swip will introduce **backward incompatibility** in the code. - +Due to secondary addresses, the new chunktype we define, as well as parameters relating to insurance requirements within a core protocol, implementing this swip will break the **backard compatibility** with earlier vrsik . ## Copyright From 3de1bf9e6ed27c7950dda8f94f1e2ade5a0e767a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20Tr=C3=B3n?= Date: Fri, 4 Jul 2025 14:01:46 +0200 Subject: [PATCH 6/6] Update swip-35.md --- SWIPs/swip-35.md | 35 ++++++++++++++++------------------- 1 file changed, 16 insertions(+), 19 deletions(-) diff --git a/SWIPs/swip-35.md b/SWIPs/swip-35.md index a40a615..bed5c17 100644 --- a/SWIPs/swip-35.md +++ b/SWIPs/swip-35.md @@ -43,11 +43,11 @@ Lack of clarity regarding the interpretation of batch parameters has led to misg #### Privacy -The signature on a stamp serves as *proof of relevance*[^2] when it comes to evidence submitted as part of the claim transaction. As long as stamps are self-signed by the uploader, the batch owner can be identified as the originator of the data for 3rd parties. +The signature on a stamp serves as *proof of relevance*[^2] when it comes to evidence submitted as part of the claim transaction. As long as stamps are self-signed by the uploader, the batch owner can be identified as the originator of the data by 3rd parties. [^2]: The validation of a batch owner associating a chunk with a postage batch serves to proof that someone is indeed willing to pay for the data in question. -Somewhat independently of this, the fact that same-owner batches are used when uploading a file, the chunks uploaded during one session leak co-constituency, i.e., which chunks are likely to constitute one file, directory or belong together in a single upload session. This overshadows somewhat Swarm's otherwise rather strong privacy claims. +Somewhat independently of this, the fact that same-owner batches are used when uploading a file, the chunks uploaded during one session leak co-constituency, i.e., which chunks are likely to constitute one file, directory or belong together in a single upload session. This somewhat overshadows Swarm's otherwise rather strong privacy proposition. #### Insurance @@ -120,13 +120,13 @@ Double signing of a slot (though detectable[^17]) will only be avoided if the ow ##### Postage batch utilisation The quota represented by the set of storage slots of a batch is evenly distributed in a number of buckets, as it also serves to force balanced use across neighbourhoods. -Even though the most effecient utilisation of a batch is if stamps can max out each bucket, already a single bucket filled up will make an immutable batch practically not usable since, as it has no slots available in the bucket filled, it cannot be guaranteed to stamp just any chunk.[^31] +Even though the most effecient utilisation of a batch is if stamps can max out each bucket, already a single bucket filled up will make an immutable batch practically not usable since, as it has no slots available in the bucket filled, it cannot be guaranteed to stamp just any chunk[^31]. -[^31] Incomplete utilisation can be counteracted with making sure you can reassign the -'same' chunk to another bucket by either [mining]() or having availability guaranteed by a constant influx of chunks (see section below). On the other hand, with the variance limited, the actual discount can also be made explicit in the current system, and, therefore [user expectations of their quota purchased that is suggested bt the batch depth must be modulated by factoring in the packed address chunks needed to represent files, and the manifest nodes to serve as directories, erasure code parities +[^31]: Incomplete utilisation can be counteracted with making sure you can reassign the +'same' chunk to another bucket by either [mining]() or having availability guaranteed by a constant influx of chunks (see section below). On the other hand, with the variance limited, the actual discount can also be made explicit in the current system, and, therefore [user expectations of their quota purchased that is suggested by the batch depth must be modulated by factoring in the packed address chunks needed to represent files, and the manifest nodes to serve as directories as well as erasure code parities, etc. Instead the volume called [*effective utilisation*](https://www.overleaf.com/4919957411cgrncysjqrmv#3b42ca) must be given to users. -The effective batch utilisation rate increases with overall volume implying a natural overhead for small batches[^1] +The effective batch utilisation rate increases with overall volume implying a natural overhead for small batches[^1]. [^1]: or more naturally put, there is an incerasing discount rate when purchasing larger volumes. @@ -178,8 +178,6 @@ We propose that chunks have a *secondary address*, defined as the (legacy Keccak So when nodes swear on to provide insurance (or insurance/availability level X service), they register their address with a stake on a (the?) staking contract. It is important that providers should be distributed evenly in neighbourhoods, ideally of the same size as that dictated by the reserve. -Maybe all full nodes are providers by default. This is very useful because when a node provides the service, they expect chunks to their neighbourhood by the secondary address (the hash of the BMT address) of a CAC --- from now a *new chunk type*. Once they receive it, they must provide the uploader with a storage receipt as an acknowledgment which they send back to the originator using the same route (backwarding). - Some insurers may just take on the task of keeping the chunks retrieveable. For this, they need to keep a postage batch alive that issued the stamp of the chunk, throughout the period matching the requirement in the chunk request. They can keep the chunk in this batch retrieveable by regularly topping up the batch if needed. Since the hashes are uniformly distributed, this batch is expected to be naturally balanced as it should be. So it is quite realistic for the node to be able to fill it. @@ -218,7 +216,7 @@ The SWEAR contract allows nodes to register their public key to become accountable participants in the swarm. Registration involves sending the deposit to the SWEAR contract, which serves as collateral in case the terms that registered nodes "swear" to keep are violated (i.e. nodes -do not keep their promise to store). The registration is open-ended Users of Swarm should be able to count on the loss of deposit as +do not keep their promise to store). The registration is open-ended. Users of Swarm should be able to count on the loss of deposit as a disincentive against foul play for as long as the insurer node has open commitments. Therefore, even a well-performing Swarm node is only eligible to reclaim their deposit after all their commitments are closed or taken over by new nodes. @@ -232,33 +230,32 @@ When a user attempts to retrieve insured content and fails to find a chunk, they can report the loss by submitting a challenge analogously to a court case in which the insurers the defendants who are guilty until proven innocent. Similarly to a court procedure, public litigation on the blockchain should be a last resort when the rules have been abused despite the deterrents and positive incentives. The challenge takes the form of a transaction sent to the SWINDLE -contract, in which the challenger presents evidence implicating a particular node having made a promise to insure that particular chunk for a period that has not ended yet. +contract, in which the challenger presents evidence implicating a particular node having made a promise to guarantee availability of that particular chunk for a period that has not ended yet. Any node is allowed to send a challenge for a chunk as long as they have evidence available to back the claim. The validity of the challenge as well as its refutation must be easily verifiable by the contract. The contract verifies challenge by checking the following conditions: -1. *covered* -- The chunk in question is (implicitly) covered by the insurance. -1. *authentic* -- The insurance was signed with the key the node registered with. -2. *active* -- The expiry date of the insurance has not passed. -3. *funded* -- Sufficient funds are sent in the transaction[^13] +1. *covered* -- The chunk in question is (implicitly) covered (in scope for the guarantee). +1. *authentic* -- The guarantor (the key used to sign the guarantee) is the identity node registered with. +3. *active* -- The expiry date of the guarantee has not passed. +4. *funded* -- Sufficient funds are sent in the transaction[^13]. [^13]: in order to compensate the insurer for uploading the chunk in case of a refuted challenge. The payment sent with the transaction is to disincentivise frivolous litigation, i.e. bombarding the blockchain with bogus challenges and potentially causing a DoS attack. -Upon succcessful challenge, an event is logged by the contract to notify the accused of their obligation to refute. The challenge remains open for a fixed time period, the end of which is the deadline for refuting the challenge. Refutation itself consists of uploading the entire chunk. This (1) serves as 3rd party verifiable proof that they keep their promise as well as (2) makes the disputed data available to nodes that require it. +Upon succcessful challenge, an event is logged by the contract to notify the accused of their obligation to refute. The challenge remains open for a fixed time period, the end of which is the deadline for refuting the challenge. Refutation itself consists of uploading the entire chunk. This (1) serves as 3rd party verifiable proof that they keep their promise as well as (2) makes the disputed data available to the nodes that require it. Upon successful refutation, the challenge is cleared from the blockchain state. While the stake of the accused node remains untouched, the cost of uploading the chunk must be borne by the challenger and is paid from the deposit. If the deadline passes without successful refutation of the challenge, -then the charge is regarded as proven and the case enters into the en- -forcement stage. Nodes that are proven guilty of losing a chunk must lose +then the charge is regarded as proven and the case enters into the enforcement stage. Nodes that are proven guilty of losing a chunk must lose their stake. Enforcement is guaranteed to be successful by the fact that nodes' stakes are kept locked up (in the SWEAR contract). If, on litigation, it turns out that a chunk (that was covered) -was lost, the deposit must be at least partly burned.[^4] +was lost, the deposit must be at least partly burned[^4]. [^4]: Note that this is necessary because, if penalties were paid out as compensation, it would provide an avenue of early exit for a registered node by "losing" bogus chunks that had been challenged on by colluding users. @@ -317,7 +314,7 @@ The will be detailed in the individual SWIPs for the components. ### Backward compatibility -Due to secondary addresses, the new chunktype we define, as well as parameters relating to insurance requirements within a core protocol, implementing this swip will break the **backard compatibility** with earlier vrsik . +Due to secondary addresses, the new chunktype we define, as well as parameters relating to insurance requirements within a core protocol, implementing this swip will break the **backard compatibility** with earlier versions. ## Copyright