The reason why we have published this before it is finished is because we want to be really open in all regards.
We want to attract thinkers to discuss all the flaws we may encounter in the hard road to make this incredible and big project to success.
Please contact us at http://talk.bitcloudproject.org
Bitcloud is a new protocol based on the ideas of Bitcoin, Mediaglobin, and Tor.
Users adhering to this protocol will be able to:
- Become an author and upload any file to the Bitcloud net and make it available to anyone.
- Download and stream the content from other users and publishers.
- Select which publisher s/he want to use.
- Authors may earn money from paid content. Free content is possible.
Nodes adhering to this protocol will be able to:
- Generate and earn profitable income in relation to the bandwidth they share.
- Select what content they cache and serve by selecting publishers.
- Be paid for guaranteed storage.
- Store non-guaranteed content based on popular acceptance.
- Be part of the government of the entire net.
- Receive a commission for inserting ads in the content.
- Rent itself to a publisher.
Publishers adhering to this protocol will be able to:
- Approve and categorize user content.
- Sell paid content and receive a commission.
- Trade with nodes for storage and bandwidth.
- Earn profitable income throughout advertising.
- Accept free content at will.
- Sell storage for the nodes and receive a commission.
Any person can become a user, a node and/or a publisher at any time. In addition to this, there will be client applications which serve as an interface to the general protocol named Bitcloud. We are presenting Wetube as the primary interface for our first version, but programmers can make any kind of interface, providing that they follow the rules described in this protocol.
The Bitcloud blockchain can be independently accessed for money transactions, so external wallets without using the main interface are perfectly feasible.
Bitcloud works by following a special set of premises referred as laws (BCL). The laws in the software ensure that the system is running smooth without spammers and selfish serving, so everybody can trust the system. These laws are initially imposed by the programmers and can dynamically change over versions.
Our original idea is to make Bitcloud available for general use without requiring special tools or software. The first interface for Bitcloud will be Wetube, which is a collection of scripts running in the browser.
In addition, bandwidth is free for users.
This will allow a whole new possibilities for mainstream adoption not possible with actual p2p protocols. Just to cite some:
- Embeddable in webpages by the insertion of a JS script acting as a very thin layer between users and nodes.
- Accessible from external languages, and easy integration with any kind of program.
- Mobile applications that work out of the box without wasting bandwidth and resources.
- Rapid access without having to earn rights like in kademlia and bittorrent.
Bitcloud separates the layer of selecting material from the storage itself, and allows nodes to select which publishers to adhere.
Because nodes are totally free to choose publishers, determining who is storing the content in the long term becomes a very fuzzy thing.
In addition, a node can also include protected content (the source and destination cannot be determined) which they can serve at the same time than unprotected content, so only direct access to the node can conclude into liability. But because source and destination cannot be determined, finding such a node becomes extremely difficult. This improves protection compared to Tor.
Bitcloud is constructed by the integration of an inter-dependant chain of modules and plugins, each one having its own BCL (laws).
For example, there are the fundamental modules of routing, storage, and goverment. From there on, a complete set of plugins can be constructed.
Each plugin will allow publishers to create sub-nets. In the case of the cloud module (the first and most important one) the publishers create a sub-net with only the content moderated by them. Collaboration between publishers is possible.
Examples:
- A social plugin, like Facebook, Twitter, and G+. Any publisher will have an independent social sub-net, so it is possible to create different social nets each one specialized under the rules of the particular publisher.
- A wiki plugin. Each publisher will have and moderate its own wiki.
- A forum plugin. Each publisher will moderate and establish the rules of his own forum.
- A market plugin. Each publisher will be in charge of one store or collection of stores and will moderate sellers and customers.
The possibilities are endless. We can virtually replace every single major layer of the actual Internet in a decentralized way.
The entire system can be fully scriptable in real time using simple to use and parse lambda expressions, so advanced users and programmers can tweak and modify the way in which they are connected to the Bitcloud net, embed actions into web pages, use in their internal programs (in any language), etc. Nodes can be accessed in real time by their owners with full control to the entire running system. We are looking forward an incredible level of flexibility in where all users can review and enhance the living (running) network.
- The Bitcloud Cryptographic Law (BCL) is a series of “laws”, and there are “judges” in such a way that workers in the net judge other workers and emit “verdicts”. Consensus in the verdicts enforce the appliance of the law in the form of rewards and penalizations, and store them in the Blockchain and Node Pool.
- The Bandwidth Law is the law ensuring that nodes route traffic from users to the Bitcloud net to get/post content. Nodes are paid for doing so, by generating money.
- The Dispersion Law ensures that no one tries to cheat the system. It does so by dispersing the connections from user to nodes and nodes to other nodes. See Proof of Bandwidth separate paper.
- Storage Law ensures that nodes store guarantied content.
- Service law ensures that nodes don’t deny service.
- We use our own language for control commands and objects.
- Money transactions are stored in the blockchain.
- Node states and verdicts are stored in the Node Pool.
A worker is basically any entity that is a user, a publisher or a node. We encapsulate this into a general class because all workers have some variables in common, like the alias or the public-key. Then we derive the specifics workers from this class.
Is usually the name or nick chosen by the worker. _TODO We have to think if we want to implement our own ID system or use an existing one like Keyhotee or Namecoin.
The private key of the worker used to sign actions. By design the worker never divulges its private key.
Is generated from the private key and is used for all wallet operations plus to verify signatures in the judges and enforcement (see below).
Is t when the enforcement routine determines so. A banned worker cannot use the Bitcloud for a specific period of time defined in banned-time.
Is t if the worker enters into a special mode in which all the content it uploads, moderates or stores is encrypted and enrouted protected (the IP cannot be determined). More below. It is possible to use both a protected and an unproteced worker in the same IP.
The list of IPs of the worker or NULL if protected.
Nodes are the primordial base for the Bitcloud network and most of the work is done there.
Nodes run the program bitcloud. This program maintains a cache of files served and provides the necessary bandwidth for the Client Applications to work.
Nodes also do the important work of providing Proof of Bandwidth so the cloud runs consistently and everyone generates the money they only deserve.
Like it happens with Bitcoin, a block reflecting all the transactions are generated once every 10 minutes. This block represents the consensus of the Bitcloud net about the operations made in itself since the last block. It contains all the money transactions made, including the ones generated by the nodes by bandwidth sharing. In order to preserve space, it does not contain all the details of the nodes states nor any content at all.
The blockchain is the chained collection of all blocks generated by the Bitcloud network. It’s contents are perpetual and immutable.
In addition, there will be a Node Pool. This pool is regenerated every 10 minutes after a new block is generated and contains the state of all nodes. When regenerated, all nodes apply the rules under the section BCL to it, to ensure that every node meets the rules and no one tries to take advantages over the rest.
The nodes store the content in the form of a cache pointing to publishers, as defined in the variables.
- Publishers have a list of contents and/or users to which they are attached.
- Nodes select a list of publishers they want to be adhered to.
- Nodes store the content for the publishers
Publishers can’t ever select Nodes. See Publisher Class for more details.
Note: anyway we can implement a kind of query that a publisher can send to a node asking for storage in exchange of money, and the node is free to accept or deny.
Are the variables that define how much bandwidth the owner of the node wants to share, expressed in megabyte/s.
Is expressed in megabytes and it is the maximum size to allocate for the files cache. When the space occupied by the cache pass this variable, the least requested files are deleted.
Refers to the public key of the node, used to sign transactions in the BCL routines.
Is a list of accepted coins for exchange. For example `(BTC LTC)`. If a node accepts BTC, it must run the bitcoind daemon.
Is a list of normal publishers for which the content is accepted to be cached. The content cache can vary, so a node can automatically select the content to be cached in function of the number of requests. If this variable is F, the node becomes unmoderated and caches everything that goes through.
Is a list of publishers whose content is always going to be cached entirely. This is useful for publishers that also own nodes.
A list of publishers for cloud personal storage. These publishers put a price for content storage. This content is guaranteed to be stored until 1 year has passed since the user paid, or until the user deletes it or the publisher stops supporting the user.
Note: the sum of the space required by the publishers in cloud-publishers and preferential-publishers should be less than max-space. bitcloud daemon should log a warning if that happens, and POS policies could ban the node.
A list of contents uploaded from the users that are waiting for the approval of the publishers adhered to this node.
How much time this node is going to wait for the approval of any publisher adhered to this node. Once that time has passed, the awaiting content is deleted from the cache. Minimum time allowed is 12 hours. Defaults to 48.
When the node will be shut down automatically. Useful for protected nodes that serve delicate content.
Whether the node IP is protected and the content in the local cache is encrypted with the private key of the node. But this has some downsides, the most important is that data is routed throughout many nodes, adding costs in bandwidth and latency.
Ćollection of IPs corresponding to this node. A cluster of nodes can be possible and load balancing of the bandwidth and storage is a must. A simple node will have only one IP. This variable is NULL for protected nodes.
Publishers are crucial to the well-working of the Bitcloud. Their main tasks are:
- Approve user content. The user selects a list of publishers to which s/he wants to adhere, the publisher receives the request and decides if the user meets his/her requirements.
- Categorize. A publisher properly corrects tagging of the content s/he receives from the user.
- Advertising. Advertisers select which publishers they want to adhere to, and ads are only shown in the content approved by that publisher. Publishers are paid for this, and they set the price. Part of this revenue goes to the publisher.
A vital and important imposed limitation: publishers cannot choose which nodes to adhere, but nodes can choose any publishers they want. This is a way to avoid censorship because any node can store any content that the owner wants. This is also a good way for nodes to make money without having to be approved by anyone. An advantage for publishers: they don’t have to have the technical skills to maintain a node, and do everything from the user interface, as the maintainer of the node can be another person, or even a rented pre-configured node from an ISP.
This is a list of contents that the publisher has been approved. Nodes who have selected this publisher have the obligation to maintain a cache with the files this publisher approves.
Users are the ones that obtain the benefits from the services of nodes and publishers, because they are who send and receive the content.
Usage of the system must be transparent to the user. S/he should be able to connect using his/her client or browser without having to do anything special and without requiring any technological expertise.
All the workers need an alias.
There are 2 kinds of aliases:
- Anonymous, it is the same as its public key, generated from a private key that only the worker knows. Length must be checked to ensure randomness.
- Registered. The worker pays a quantity for a name, and that name is attached to his public key in the Blockchain forever.
Every worker owns a private key. A public key is generated from the private key.
The private key is never sent in any way. The worker is responsible for all the security matters of managing this key.
The public key of nodes working for the current period is published in the Node Pool.
Money transactions affecting the public keys of any worker are published in the Blockchain.
Bitcloud will allow to pay services in Bitcoin (and probably other currencies), and for that, a system of automatic escrow must be provided.
See Escrow Law
Most communications between users and nodes are done using the well understood and easy to use HTTP network protocol as a base. We choose to use HTTP because it is enough for our requirements and because it is widely available.
We choose the default 80 port for communications, with the option for nodes to select any other alternatives. The reason to use this port is that people behind a firewall can still access the complete net without the typical restrictions. Most ISPs prioritize this port over the rest, making it the fastest and more reliable port. Also, censorship becomes much more difficult.
Nodes wanting to use the 80 port for other purposes (like for example a classical web server) can use any of the Unix tools available to route domains, like for example nginx.
We don’t use any p2p low level protocol like Bittorrent because we need very fast access with the least possible lag. It also happens that many ISP actually censor Bittorrent protocol. In addition, using simple HTTP will simplify the protocol very much.
We are also supporting the serving of static web pages stored in the nodes outside of the Bitcloud, in the official Bitcloud server program. These pages are considered a wrapper for Bitcloud. The purpose of this is to allow nodes to become entry points for users:
- To allow the users to download a full featured client.
- To allow the users to connect “hot” using a JavaScript client.
- To allow nodes to provide informational static content only relevant to that node. For example, a TV channel, a book editor, a corporate web. They can embed content from Bitcloud in those static pages.
- To publish the statistics of the node in real time to the WWW.
NOTE: those static pages are considered to be out of the Bitcloud, except if the content is embed (and only for the content), so they don’t apply to the law of Bitcloud, nor to its benefits.
In addition to this, nodes can install their own HTTP server (like Apache) and serve dynamic pages using local routing facilities like nginx.
We don’t need to resolve node names because content is provided to the users by simply requesting it to the nodes s/he connects to.
Each content is signed by a user and a publisher. The workers only have to request the content matching the hash of the content, and the routing routines are in charge of finding the nodes having it.
Every new node must register into the network and provide the necessary details specified in the Node class.
Every user is connected to a randomly selected collection of 4 nodes. Dispersion Law.
The user running a full featured user client can find those 4 nodes directly. Typically the client will have a collection of trusted nodes already coded in, and they can obtain the rest of the available nodes from any of them.
The users running a thin client in JavaScript in the browser do obtain the script from an Entry Point.
Both clients act as full workers and are governed by the laws. If a user does use a corrupted client that tries to cheat the system, he is penalized and/or banned by applying all the laws affecting users. Service Law.
See also in the separated paper:
https://github.com/wetube/bitcloud/blob/master/proof-of-bandwidth.org
Every single user is connected to 4 nodes using the User Router coded in the interface. These connections are meant to both transfer the content and to apply the BCL (Bitcloud Cryptographic Law).
Even if the content is not protected, all communications are encrypted, in order for an external viewer to not be able to distinguish between protected and unprotected content.
Encription in the server end is also recommended.
Example graph:
The process (using the graph above as an example):
- The User Router finds 4 random nodes. This router is coded in the user interface. Dispersion Law
- The User Router asks the 4 nodes to find a route to the content. Service Law
- Nodes answer with route data. Service Law
- The User Router selects the fastest. The fastest is calculated in function of the data retrieved in the last point, and usually is a combination of bandwidth available and shortest distance. In the example, Node3 is the fastest because it happens to be also a Final Node for the requested content. The second fastest should be Node2 because it has more connections to nodes having the content. If the requested content is cached in any of the Nodes, it should be considered as fast as actually being a Final Node.
- Money is generated only in the User Router’s connected nodes, never in the Final Nodes to avoid short circuits. Only exception is when the Final Node is also one of the connected nodes. Bandwidth Law
- If a Final Node denies service, Service Law is applied.
- Optionally, connected nodes can cache the content so they don’t need to retrieve the content from the Final Nodes if the content is requested often.
- While the User is downloading/uploading the content, he must inform all the connected nodes about the quality of the connection. If quality is low, he can select another route from another node and denounce about the issue to the other nodes, so they can apply the law. See “User Router” section for a better explanation. Bandwidth Law
- If quality is low, an Investigation Process to find the culprit is executed. Culprit can be the Final Node, the connected Node, both, or none. Bandwidth Law - Check low quality culprit section.
- User is banned if he tries to abuse bandwidth or the law. Bandwidth Law - Abuse check section.
- Law is enforced when at least 3 of the 5 workers agree (the 5 workers are 1 user + 4 nodes).
See also the separate paper:
https://github.com/wetube/bitcloud/blob/master/proof-of-bandwidth.org
When a content is marked as protected, the Bitcloud net protects both the IP and domain name of the node serving the content. The user requesting the connection is also protected. There is still a way for auditors to know that a certain computer is acting as a protected node, but will not be able to know what is the protected content that is serving. Even more, external auditors cannot distinguish between the content routed or served while analyzing the traffic to a specific node.
All nodes are enforced to accept protected content routed, even if they are not protected themselves. Nodes cannot know what the content being routed is because it is multi-layered encrypted.
We use a routing process similar to Tor Onion hidden services, optimized to our purposes.
- The user chooses a collection of 4 random nodes. Dispersion Law.
- The User Router (coded in the interface) asks the 4 nodes to find a route to the Protected Node, encrypting the request with the public key associated to the Protected Content, and adding its public key inside the encrypted request (so only the real Protected Node can know what is sent back). Service Law
- Each node tries to find a competitive route to the Protected Node and answer fast to the User. The way to do this is by sending the encrypted request to all the nodes it is connected to and wait for an answer. Service Law
- The Private Node is found when it is able to decrypt the request. Otherwise it is re-dispatched to all connected nodes.
- To protect itself, the Protected Node adds a random time (lag) of response to the request, equivalent to several searches. We recommend a random number between 1s and 5s. Once answered, the node must stick to that approximate lag, but can change it when requesting a different content for another publisher or when certain time has passed.
- There is the lucky possibility that a connected Node is also the protected one having the content (like Node1 in the graph). In that case the protected node adds the convenient lag time to protect itself.
- Meanwhile all nodes are attending other requests for other contents, so nobody knows if the other requests are actually part of the original request or not.
- The Protected Node always sends back the content encrypted with the public key inside the request, so only the user can know what the content is.
- The User Router chooses the fastest route.
- Only the User Router’s connected nodes generate any money, to avoid short circuit cheats. Bandwidth Law
- The User download/stream/upload the content throughout the fastest route. Bandwidth Law
- Meanwhile, the User informs the rest of the nodes about the quality of the connection. Bandwidth Law.
- If the quality is low, the User Router tries to change to another route Node while informing the issue. Bandwidth Law
- The selected node also enforces the Bandwidth Law and Service Law for the Middle Node it is connected to.
- Each Middle Node also enforces the Bandwidth Law for their connections.
The mission of the User Router is to find the fastest route to a solicited content, whether it is protected or not.
It must be coded in the user interface and must respect the BCL in order for the user to not be banned.
Its principles are simple, and should be easy to code in any language, even in JavaScript for the browser.
The Cryptography Law, also called Proof of Stake in more simple projects, is the law of Bitcloud - what governs it - providing the rules in the form of algorithms executed by the nodes and clients in real time.
Each rule must be applied for every node or client in the net. Compliance to every rule is rewarded in the way described by the rules. Non-compliance results in penalization.
This fine equilibrium ensures that every node does the work correctly and there are no abuses.
Like in the real world, here there are laws, judges and rule enforcers. The main difference is that here all nodes and clients are the judges and the rule enforcers, so the law coming in the form of computer algorithms is applied by the consensus of the entire Bitcloud net.
We have named every rule with an easy to remember name, so workers of the system can refer to them when talking to other workers and non technical people.
Basically, the law is applied by judging (checking) that every node and client is doing the work as it should, so, when asked, it should answer with the truth of what is asked. If it is found that the node or client is lying, it is penalized or banned, and its transactions rejected are not included in the blockchain.
Laws are written in the source code in the form of generics and the corresponding methods. A method is a specific application of a generic. For example, for the generic of the Law of Bandwidth there are going to be several methods for judging nodes, users and publishers.
This is a subprocess of the general sync process, as described in the Nodepool section.
The return of those functions is always a verdic.
The purpose of it is to certificate (sign) the verdict and write it in the Node Pool State when appropriate. When there is general consensus (for example, 80% or more of the workers agree with the verdict), the current Block is frozen and added to the Blockchain, and the Node Pool State is regenerated using relevant parts of the old data for the new start.
- All nodes must be in sync of time with the official world time at GMT with a maximum variation of 10 seconds. Clients are not forced.
- Check is made sending a request to the other node and comparing to the actual date in the moment of sending the request. Function returns `(get-universal-time)` of the other node.
- No reward for compliance.
- Penalization of 1 MAC (X) for every 10 seconds out of sync on each block generation. Ban for Y periods if it is 30s or more or the node hasn’t got enough money to pay penalization. All operations of the node for the current block are rejected (Z) only if time variation is bigger than 30s. Y is 1 for one time ban. Y is 6 if the node persists. Y is 144 if it persists even more. Y (1,6,144)
- All nodes must write the correct dates in all connections and associated transactions.
- Check is made looking at the states written by the node in the pool. All dates must be correctly dated within the period of the block being analyzed.
- No rewards for compliance.
- Immediate penalization of 1 MAC and ban for Y (6, 144, 288). Reject all transactions.
- All workers must answer in less than X seconds to any query or sync function. Recommended constant for now: 600ms.
- No rewards for compliance.
- Ban X(1,2,6,144) if slow or saturated connections.
In a separate paper for now:
https://github.com/wetube/bitcloud/blob/master/proof-of-bandwidth.org
DRAFT: Workers must answer to queries exactly as they are supposed to answer. The worker informs to all other connected workers about how things are going on in its relation to the connected worker. Verdicts are written in the Node Pool as always. If it is found that a node denies service to another worker, penalization happens. No reward for compliance.
In a separate paper for now:
https://github.com/wetube/bitcloud/blob/master/proof-of-bandwidth.org
Taking in count that nodes can select publishers, and several nodes may store the content of the publishers, that means that there must be a proper way to ensure:
- Efficent storage distribution.
- That nodes cannot mess with the data, and a way to check it.
- Fast access to the data.
- Proper protection for the content.
Being discussed here: http://talk.bitcloudproject.org/index.php?topic=10.0
Bitcloud allows the interaction with external currencies like Bitcoin.
Bitcoin allows escrow since version 0.9. https://en.bitcoin.it/wiki/Contracts
The only problem with the escrow in Bitcoin, is that there must be a third party (oracle) who is reviewing the work and signs the release or refund of the escrow once the work is done.
In our system, there cannot be a trusted central authority because that defeats the idea of full decentralization. Also, we cannot trust individual nodes to do the work, because some of them might not be available and others could act maliciously.
So we propose a solution: using Shamir’s sharing key algorithm to protect the key that releases the escrow, by distributing that key among several well reputed nodes. This way, when an order is made in the Sync process to accomplish the release or refund the escrow of a transaction, at least half of the nodes having the Shamir encoded key must participate in the event.
This is a new system of automatic escrow that can be entirely run in a distributed way with little to none possibilities of defeat, providing that nobody possesses more than 50% of the total well-reputed nodes.
When a customer buys a service to a node or a publisher, a new escrow key is created by the customer and the connected node. Then, the connected node encrypt it by using the Shamir’s algorithm and distribute the encrypted keys along the best nodes (those with no recent penalization and a good stand up time).
When the work is done, the implied nodes are judged by the different applicable laws (ex. Bandwidth Law or Storage Law) while the sync happens. If the verdict is favorable, an order to the nodes having the encrypted key is sent. All nodes having this key put the parts together and obtain the secret key to release the escrow, and do so.
Service law is applied to ensure that each part does the work.
We use a Blockchain in order to store all money transactions, including the ones coming from the nodes for Bandwidth sharing.
We are not using the Blockchain to store any information related to the content at all. If we do so, very soon the Blockchain would become very big. For such things, we use the Node Pool.
We can’t use an existing Blockchain from another coin like Bitcoin for the simple reason that we must generate the money from bandwidth sharing. But we are providing exchange facilities that could be inserted in a user interface like Wetube.
A new block is generated once every 10 minutes after the Node Pool is synced and signed.
In order to avoid some scalability issues, there is a process of simplification that happens for data stored in the blockchain from certain time backwards. This process removes all transaction data except the amount of money owned by every account when due-date arrives.
The Nodepool contains all the information relevant to the nodes, and it is constantly calculated in real time.
This is the interface for the node (not the users) so the administrator can see and tweak values in real time.
Bitcloud is a very complex distributed database and certainly cannot be satisfactory operated without a proper language.
We choose a similar query language to SQL, but using s-exprs or lambda expressions so it is easier to parse, use and expand. We provide many convenient forms and functions so user interface designers can extensively take advantage from them.
Any command is visually constructed by first specifying the name and then their parameters, all between parenthesis, this way:
(command value1 :param2 value2 :param3 value3)
All parameter names are prefixed with a :.
The first value doesn’t need a parameter name because it is the fundamental parameter for the command.
The rest of the parameters are named, and in fact their positions doesn’t matter at all.
Values can be of type:
- strings, surrounded by ””:
(search "wetube") - numbers inserted directly
(search "wetube" :show-results 50) - single strings or symbols, they are the same as strings but used for
just one word:
(search 'wetube) - literal lists prefixed by a ‘ and followed by parenthesis and values
separated by spaces:
(search 'wetube :publishers '(modA modB)) - booleans which are just t or f:
(verdict t :reward 1.23) - output from other commands surrounded by parenthesis without the ‘:
(search 'wetube :publishers (search-publishers 'mod :protected t))
In addition to commands, we also use lambda expressions to form the format of objects, mainly used for return values from commands, blockchain transactions and Node Pool states.
(object value1 value2 :slot1 slot-value1 :slot2 slot-value1)
In contrast to commands, an object may have any number of unnamed values. It can be the case that some objects just don’t have any predefined slots.
Named slots are always at the end of the lambda list to avoid confusion.
It is better seen as examples:
(verdict t :reward 1.23) (results )
We provide many utility functions and constructs so workers can do very advanced queries to nodes.