In 2007, Ateniese et al. [8] first proposed a Provable Data Possession (PDP) protocol that can verify the integrity of cloud data. This scheme uses a probabilistic strategy to complete integrity verification, while using RSA Homomorphically Verifiable Tags (HVTs) to aggregate evidence into a small value. This not only reduces the computing overhead of cloud storage, but also greatly reduces the communication overhead of the protocol due to the characteristics of signature aggregation. In 2007, Juels et al. [9] proposed another classic verification scheme, POR scheme. The scheme effectively identifies the damage of the outsourcing documents by implanting some “sentinel” checking data blocks in the outsourcing documents and uses the Reed-Solomon error correction code to perform fault-tolerant preprocessing on the outsourced files, so as to restore the damaged data files. However, in this scheme, the data owner needs to consume huge computational cost for erroneous data recovery and original data encryption. In 2013, Yang et al. [10] proposed a cloud data privacy protection protocol, which can better solve the security risk of data confidentiality in the public audit process in Compact Proofs of Retrievability (CPOR). The protocol improves the overall performance of the scheme by reducing the number of data tags by using data fragmentation technology and HVT. In 2017, Hiremath et al. [11] introduced an efficient data auditing method that uses the AES encryption algorithm and SHA-2 (Secure Hash Algorithm), utilizing a third-party auditor to perform integrity checks. In this method, the user encrypts the data using the AES algorithm and obtains a message digest of the encrypted data using SHA-2. Encrypted data is sent to the cloud server, and message digests are sent to a third-party auditor (TPA) that performs data integrity checks. In 2018, Han et al. [12] proposed a pairless integrity verification scheme based on Schnorr signatures. However, this scheme suffers from computational errors in the domain and requires a third party. In 2019, Zhang et al. [13] proposed a general construction method for PoR based on Linear Homomorphic Structure Preserving Signatures (LHSPS). The unforgeability of LHSPS ensures the authenticity and tractability of the PoR scheme. In 2020, Yu et al. [14] designed a more efficient pairing-free scheme based on blockchain. However, this scheme can only be applied to private audits.

In 2008, Satoshi Nakamoto first introduced it into the blockchain in his paper [15], which is the underlying technology of Bitcoin. Traditional transactions require a centralized and trusted institution. The confirmation and recording of transactions are completely dependent on trusted institutions, which can lead to many issues such as transaction costs, efficiency and security. PoW is the consensus algorithm used in Bitcoin. Its core idea is to allocate accounting rights and rewards through the competition of computing power among nodes. Based on the information from the previous block, different nodes calculate a specific solution to a mathematical problem [16]. This math problem is difficult to solve. The first node to solve this math problem can create the next block and be rewarded with a certain number of bitcoins. The earliest application of PoS is PPCoin. In PoS, digital currency has the concept of coin age. The coin age of a coin is its value multiplied by the time period since it was created. The longer a node holds coins, the more power it has in the network. Coin holders will also receive certain rewards based on the coin age. In the design of PPCoin, mining is also required to obtain bookkeeping rights. The formula is proof hash < coin age * target. The proof hash is the combined hash of the weight factor, the unspent output value, and the fuzzy sum of the current time [17]. With the concept of coin age, blockchains no longer rely solely on POW. This effectively solves the resource waste problem in PoW. BitShares is an example of DPoS [18]. In a blockchain with DPoS, each node can choose witnesses based on their stake. In the entire network, the top N witnesses who participate in the election and get the most votes have the right to keep accounts. The number of witnesses, N, is defined as at least 50% of voting stakeholders believe there is sufficient decentralization.

In 2014, Miller et al. proposed a new scheme, Permacoin [19], which modified Bitcoin to produce highly decentralized file storage and reduce the overall waste of Bitcoin. Permacoin encourages participants to store locally, ensuring a high probability of complete data recovery, enabling robust file distribution. In 2018, Hao et al. [20] proposed a blockchain-based decentralized model called DCOM (Decentralized COllaborative verification Model). The model consists of a cooperative network of validating peers, each of which maintains a record of validation via a blockchain. Francati et al. [21] proposed a blockchain-based decentralized storage system—Audita in 2019. It can be built on multiple blockchain systems and uses an enhanced network of participants including storage nodes and block creators.

Our goal is to provide a distributed storage system with proof of retrievability and deduplication algorithms as consensus mechanisms, incentivizing a majority of nodes to participate in storage services (see Fig. 1). Because centralized storage cannot guarantee user privacy and user information is easily leaked, we use ring signatures and blockchain to achieve distributed storage. The system model consists of three entities: Data Owner (DO), Cloud Storage Provider (CSP), Third-Party Auditor (TPA).

Data owner (common node): The data owner is a node that owns a series of files that need to be stored on the cloud.

Cloud storage provider (storage node): The cloud storage provider is a node that provides cloud storage services to data owners.

Third-party auditor (retrieval node): The third-party auditor is a node that provides retrieval services for users.

The entire design includes the following functions: file storage, deduplication, privacy protection, and block election.

File storage. To store files and ensure data integrity, the data owner uses proof of retrievability and encodes the files. Then the data owner sends a storage request, agrees on a price with the cloud storage provider, and signs a contract on the blockchain [22]. Cloud storage providers accept files and are checked by retrieval nodes for the duration of the contract. The retrieval proof will be issued as a transaction, including a reference to the storage contract in the blockchain and the storage integrity proof. The retrieval node and storage node will also receive a portion of the service incentive [23].

Deduplication. In order to meet the optimization of storage capacity in cloud storage, a deduplication algorithm is added to the solution [24]. In terms of data integrity verification, duplicate identity authentication tags are not introduced, thereby ensuring that communication costs and computing costs remain unchanged. At the same time, a polynomial-time adversary without a complete data file cannot pass the verification process [25]. The algorithm greatly reduces the demand for physical storage space and brings many benefits to the entire system, such as saving the total storage cost and management cost; efficiently controlling the accelerated growth of data; increasing effective storage space and improving storage efficiency; saving network bandwidth for data transmission [26].

Privacy protection. User privacy is one of the most important aspects of privacy protection. The system model needs to effectively manage and control personally identifiable information and enhance the confidentiality of documents. To do this, we use ring signatures to construct our scheme [27].

Multiple users participate in the file storage stage. During the signing process, the public key of everyone is used to sign the file, including the storage node, which makes the file signature untraceable. Unrelated nodes in the blockchain system cannot trace the sender of the file [28]. When other nodes verify the transaction, they can only determine that the file signature is one of many public keys, but cannot locate the specific sender of the file. At the same time, Sybil attacks and outsourcing attacks initiated by some malicious storage nodes are prevented.

Election blocks. When data owners store data, they need to issue storage orders and a certain amount of blockchain tokens to reward storage nodes and retrieval nodes [29]. Storage nodes and retrieval nodes receive corresponding rewards after completing storage and retrieval services, and record transaction orders and storage proofs on the blockchain [30]. In our model, the blockchain will elect a leader based on the current storage node capacity to generate new blocks.

Inspired by the audit scheme [31], we briefly define the security model of the scheme.

Initialization: Challenger C first generates a random file and runs the algorithm to generate the file key pair (pk, sk) and the encoding block {m_i}. Then, C sends the public parameter para to the adversary A.

Query: The adversary randomly selects a block m_j ∈ {m_i} and queries the challenger C to obtain the corresponding hash value and signature until the query time reaches q_s.

Challenge: C randomly challenges A to some block Chal_t that has not yet been queried. According to Chal_t, generate aggregate signature σ_t and proof Proof_t, and return them to C.

Verification: Check that σ_t is consistent with Proof. If they agree, A wins, otherwise, A loses.

To enhance security and privacy protection, our proposed distributed storage scheme should satisfy validity, unforgeability and privacy protection. Defined as follows:1)

Effectiveness

Validity means that for any security parameter λ and a negligible function negl( ⋅ ), all proofs generated by honest nodes must pass verification, while proofs generated by malicious nodes cannot pass verification.2)

Unforgeability

Unforgeability means that no adversary to our verification scheme can make a verifier accept a proof of retrievability protocol instance with non-negligible probability unless it responds with a correctly computed value.3)

Privacy protection

Privacy protection means that the signer of the file identification in the scheme has unconditional anonymity, that is, for any algorithm A, any set of users R = pk₁, pk₂, …, pk_n, the probability P[pk = pk^′] is all 1/2, where Tσ=(Is,c1,c2,…,cs,…,cn,d1,d2,…,ds,…,dn) is the ring signature generated by pk_s.

In the scheme we have used the following symbols (as shown in Tab. 1).

We use λ to denote the security parameter. F represents the encoded file after user preprocessing, where F = (m₁, m₂, …, m_n). H( ⋅ ) represents a one-way hash function. G is the base point on the elliptic curve. Let A denotes an algorithm, then, the notation a ← A(x) denotes an algorithm A that takes an input x and gets an output a. e is a bilinear map in the scheme.

In this section, we describe the construction of the proposed distributed storage consensus protocol.

KeyGen:

Given the security parameter λ, the TA selects a random number α⟵RZq∗ to generate the public key {gαj}j=0t+1 , where α is the master key known only to the TA.

Data owner DO_i randomly selects sski=xi∈Zq∗ , calculates spk_i ← x_i ⋅ P, generates a signature key pair (spk_i, ssk_i), then selects a random number x←Zq∗ and calculates ɛ ← g^x, v ← g^αx.

Among them, PK={q,ϵ,v,spk,u,{gαj}j=0s+1} , SK = {x, ssk}, MK = {α}.

Setup:

Data owner DO_s encodes the file F, F^′ ← F, F^′ = {m_ij}, 1 ≤ i ≤ n, 1 ≤ j ≤ t − 1. Randomly chooses the filename name∈Zq∗ .

Selects a signature public key set R = {spk₁, spk₂, …, spk_n}, which contains the CSP’s signature public key for verification, randomly selects ui,vi,wi∈Zq∗ , calculates L_i from Eq. (1), and calculates R_i from Eq. (2):

(1) Li={(ui+vi)∗G2ifi=sui∗G2+(vi+wi)∗spkiifi≠s (2) Ri={(ui+wi)∗H0(spki) ifi=sui∗H0(spki)+(vi+wi)∗Isifi≠s

Among them, Is=ssks∗H0(spks) . The purpose is to prevent double spend attacks. H₀(spk_i) maps spk_i to a point on the finite field elliptic curve.

Chooses r∈Zq∗ randomly and then calculates h, c_i, e_i from Eqs. (3)– (5).(3) h=H2(name||r) (4) ci={H1(h,L1,…,Ln,R1,…,Rn)−∑i=1nciifi=sui∗H0(spki)+(vi+wi)∗Isifi≠s (5) ei={(ui+vi)−ci∗skiifi=suiifi≠s where name is the content of the signature here, and the ring signature of the file name name by the data owner DO_s is output as Tσ=(Is,c1,c2,…,cs,…,cn,d1,d2,…,ds,…,dn) . Finally, DO_s uses the signature private key ssk_s to generate the file identifier τ←name||n||Tσ .

For each file block m_i, every file block generates an authentication tag σi=(uH(name||i)⋅∏j=0s−1gmijαj+2)x=(uH(name||i)⋅gfβi→(α))x , where βl→={0,0,mi,0,…,mi,s−1} .

Data owner DO_s stores {F^′, τ, σ_i} into CSP.

Challenge:

To verify data integrity, the third party first uses spk_s to verify τ and the signature on τ. Terminate if the signature is invalid. If the signature is valid, randomly selects the pair (k₁, k₂) where k1,k2∈Zq∗ , and sends the challenge value chal = (c, k₁, k₂) to the CSP.

Prove:

CSP calculates a_i = ϕ(k₁, i), i ∈ c as the random index of the challenge block and bi=k2aimodq,i∈c as the random parameter. Then generate y=fA→(k2) , A={0,0,∑i∈cbaimai,0,…,∑i∈cbaimai,s−1} .

Then fω→(z)≡fA→(z)−fA→(k2)z−k2 , ω→=(ω0,ω1,…,ωs+1) . Next, generate ξ=∏j=2s+1(gαj)ωj .

The CSP finally computes σ=∏i∈kσaibai and sends Prf = {σ, ξ, y} to the third party.

Verify:

The third-party computes u=∑i∈cbaiH(name||ai) and η=uμ first. The user then verifies that e(η, ɛ) ⋅ e(ξ, v, κ^−r) = e(σ, g) ⋅ e(ɛ^−y, g) against Prf = {σ, ξ, y}.

Deduplication:

The CSP randomly selects k₃ ∈ Z_P, computes d_i = ϕ(k₃, i) and sends it to a third party.

After receiving D, the third party responds to the corresponding data block {mdi} , and the CSP calculates σ′=∑i∈Dσdi , η′=∏i∈DuH(name||di) , ξ′=e(∏j=2s+1(gαj)Bj,ϵ)=e(gfB→(α),ϵ) , B→={0,0,∑i∈Dmi,0,…,∑i∈Dmi,s−1} .

The CSP then verifies the integrity of the data block by e(η^′, ɛ) ⋅ ξ^′ = e(σ^′, g).

Theorem 1: If the data owner and the storage node honestly follow the proposed storage protocol, any challenge-response verification can pass the verification of the retrieval node. The correctness of the equation is as follows.

Proof: Based on Eq. (6), we obtain the correctness of Theorem 1.(6) e(η,ϵ)⋅e(ξ,v⋅ϵ−r)=e(u,g)x(∑i∈CbaiH(name||ai))⋅e(gfω→(α),gϵ(α−r))=e(u,g)x(∑i∈CbaiH(name||ai))⋅e(g⋅g)fA→(α)−fA→(r)α−rϵ(α−r)=e(ux(∑i∈CbaiH(name||ai))⋅gxfA→(α),g)⋅e(ϵ−y,g)=e(σ,g)⋅e(ϵ−y,g)

Theorem 2: If the data owner follows the data deduplication protocol, any challenge-response verification can pass the verification of the storage node. The correctness of the equation is as follows.

Proof: Based on Eq. (7), we obtain the correctness of Theorem 2.(7) e(σ′,g)=e(ux(∑i∈DpiH(name||di))⋅gϵfB→(α),g)=e(u,g)ε∑i∈DpiH(name||di)⋅e(g,g)ϵfB→(α)=e(η′,ϵ)⋅ξ′

Theorem 3: The verifier verifies the document identity T according to the formula, and if T is correct, the verification is passed.

Proof: Based on Eqs. (8)–(12) and (13), we obtain the correctness of Theorem 3.(8) ∑i=1nci=H1(h,γ1,γ2,…,γn,δ1,δ2,…,δn)

When i ≠ s, γ_i, δ_i can be expressed as Eqs. (9) and (10).

(9) γi=ei∗G+ci∗pki=ui∗G+(vi+wi)∗pki=Li

(10) δi=ei∗H0(spki)+ci∗Is=ui∗H0(spki)+(vi+wi)∗Is=Ri

when i = s, γ_i, δ_i is represented as follows:

(11) γi=ei∗G+ci∗pki=((ui+vi)−ci∗ski)∗G+ci∗pki=Li

(12) δi=ei∗H0(spki)+ci∗Is=((ui+vi)−ci∗ski)∗H0(spki)+ci∗ssks∗H0(spks)=ui∗H0(spki)+vi∗H0(spki)=Ri

Therefore, according to the above equation, the correctness of the file identification can be verified by the Eq. (13).(13) H1(h,γ1,γ2,…,γs,…,γn,δ1,δ2,…,δs,…,δn)=H1(h,L1,L2,…,Ls,…,Ln,R1,R2,…,Rs,…,Rn)=cs+∑i=1,i≠snci=∑i=1nci

Theorem 4: If the signature scheme is unforgeable and the computational Diffie-Hellman problem on bilinear groups is difficult, then any adversary to our public verification scheme for correctness cannot make the verifier accept the proof of retrievability protocol instance with non-negligible probability unless it responds with a correctly computed value.

We prove this theorem with a series of games defined in [37].

Game 0: The first game, Game 0, is simply a challenge game, similar to [37].

Game 1: Game 1 is the same as Game 0, with one difference. The challenger maintains a list of all signature tags issued as part of the storage protocol query. If an adversary submits a tag when initiating the proof of retrievability protocol or as a challenge tag, the challenger will abort if this is a valid tag that the challenger has never signed.

From the definition of games 0 and 1, it is obvious that if the opponents of games 0 and 1 have different success probabilities, we can use the opponent to construct a counterfeiter for the signature scheme.

Game 2: Game 2 is the same as Game 1, except that in Game 2, the challenger keeps a list of its responses to the opponent’s query. The challenger now observes each instance of the adversary’s proof of retrievability protocol. Suppose prf = {σ, ξ, y} is the expected response from an honest prover and Prf^′ = {σ^′, ξ^′, y^′} is the adversary’s response. The verification of prf = {σ, ξ, y} is e(η, ɛ) ⋅ e(ξ, v ⋅ ɛ^−r) = e(σ, g) ⋅ e(ɛ^−y, g), and the verification of prf^′ = {σ^′, ξ^′, y^′} is e(η,ϵ)⋅e(ξ′,v⋅ϵ−r)=e(σ′,g)⋅e(ϵ−y′,g) . Then we can know that e(ξ,v⋅ϵ−r)e(ξ′,v⋅ϵ−r)=e(σ,g)e(σ′,g)⋅e(ϵ(y′−y),g) . At this point the opponent knows that η=u∑i∈KbaiH(name||ai) . We denote ξ^′ as gθ′ , η^′ as gρ , σ′=gπ′ , according to the equation drawn above, we have e(gρ,ϵ)=e(g−y′,ϵ)⋅e(gπ′x,ϵ)e(gθ′(α−k2),ϵ) , then ρ=−y′+π′x−θ′(α−k2) , that is, (ρ + θ^′(α − k₂))x = −xy^′ + π^′. In this case, the adversary can output g(ρ+θ′(α−k2))x=ϵ−y′⋅σ′ . If the adversary knows the value of θ^′, he can get (v⋅ϵ−r)θ′⋅ηx=ϵ−y′⋅σ′ . That is, given g and g^x, where x is unknown, the adversary can solve the Static Diffie-Hellman problem with instance ux=(ϵ−y′⋅σ′(v⋅ϵ−r)θ′)(∑i∈CbaiH(name||ai))−1 . If the adversary does not know the value of θ^′, TA gives the adversary ξ′(α−k2)⋅η=g(ρ+θ′(α−k2)) and ɛ = g^x, where the adversary does not know x and ρ + θ^′(α − k₂), and the adversary can solve the CDH problem with instance ϵ−y′⋅σ′ . Obviously, σ^′ = σ.

Game 3: Game 3 is the same as Game 2, with the following differences: As before, the challenger observes the proof of retrievability protocol instance. Suppose the document that caused the abort was the signature {σ_i}, and suppose Q = (a_i, b_i) was the query that caused the challenger to abort, and the adversary’s response to that query was P^′ = (σ^′, ξ^′, y^′). Let P = (σ, ξ, y) be the expected response from an honest prover. We have already proved in game 2 that σ^′ = σ, that is, only bi′ and b_i can be different, i.e., (ξ^′, y^′) and (ξ, y) can be different. Defining Δξ = ξ^′ − ξ, Δy = y^′ − y, the simulator answers the adversary’s query. Finally, the adversary outputs a fake proof P^′ = (σ^′, ξ^′, y^′).

Theorem 5: The signer of the file identification in the scheme has unconditional anonymity, that is, for any algorithm A, any set of users R = pk₁, pk₂, …, pk_n, the probability P[pk = pk^′] is all 1/2, where Tσ=(Is,c1,c2,…,cs,…,cn,d1,d2,…,ds,…,dn) is the ring signature generated by pk_s.

Proof:

The output signature is obscured to any third party until the signer actively discloses all information. In the ring signature generation algorithm Aring, the L_i and R_i values required to calculate c_i and e_i are calculated by the signer by randomly selecting the corresponding ui,vi,wi∈Zq∗ , and the signer’s private key is also randomly selected to obtain ski∈Zq∗ . So, the result of signature Tσ is uniformly distributed in G. The probability of members outside the ring guessing the actual signer is not more than 1/(n + 1), and the probability of members in the ring guessing the actual signer is not more than 1/n, so this signature scheme complies with unconditional anonymity.

Fig. 2 shows the relationship between the number of data owners and the storage overhead between storage nodes. In our proposed consensus protocol, storage nodes keep only one file copy for duplicate data. That is, the storage node always maintains only one copy of the file, even if the number of data owners is increasing [40]. Compared with the SW scheme or other schemes, our consensus protocol has lower storage overhead.

The consensus protocol we propose has three algorithms in the audit phase: Challenge, Prove, and Verify. Concepts used in the scheme we give definitions in Tab. 2 and experimental results in Fig. 3 to visually describe the computational overhead of these three algorithms. During the integrity verification audit process, the third party runs the challenge algorithm and sends the challenge information to the CSP at a negligible cost. After receiving the challenge value, the CSP performs (k + s − 1)MUL and (s + k)EXP operations to generate the storage proof. After that, the computational complexity of the third-party audit proof is O(1)MUL + O(1)EXP + O(1)Pair. In addition, in the deduplication stage, the user does not need any computational overhead, and the computational complexity of the operation that the user needs to perform is O(s + d)MUL + O(s)EXP + O(1)Pair. In the experiments, we choose to challenge the number of blocks from 1 to 1000. When the number of challenges is 1, the running time of the Challenge algorithm is the least, which is 0.016 s, and it reaches 0.216 s when the number of challenges grows to 1000. In the Prove algorithm, when the number of challenge blocks is increased from 200 to 1000, the running time increases from 0.346 s to 3.879 s. In the Verify algorithm, the running time required to challenge 1000 blocks is 10.347 s, which takes the most time among these algorithms. It can be concluded that during the audit process, the computational cost is linearly related to the number of challenge blocks [41]. Fig. 3 shows that the computational cost of the three algorithms Challenge, Prove, and Verify varies with the number of challenge blocks.

In our proposed consensus protocol, the algorithm in the audit phase does not bring communication overhead to users. At the same time, if the storage node has a copy of the storage file, the communication overhead with the data owner will be less than that without a copy of the storage file. Fig. 4 shows the change in communication time between the number of challenge blocks from 200 to 1000, as the number of challenge blocks increases during the audit process. Scheme [38], scheme [39] and our proposed scheme all have an upward trend in communication time, and our scheme has more advantages than them in communication time.