NEAR PROJECT REPORT

Author: Gamals Ahmed, CoinEx Business Ambassador

ABSTRACT

The effects of the web by a number of companies have seduced a large number of users as these companies keep their data to prevent them from searching for alternatives. Likewise, these huge platforms have attracted applications to build their highest ecosystems before either severing access or actively opposing their interests when the applications became so successful. As a result, these walled gardens have effectively hindered innovation and monopolized large sections of the web. After the emergence of blockchain technology and decentralized cryptocurrencies, the need for applications to support decentralization has emerged. Several blockchain-based companies, applications and platforms have appeared in decentralization. In this research report, we will explain the approach adopted by the NEAR decentralization platform in designing and implementing the basic technology for its system. Near is a basic platform for cloud computing and decentralized storage managed by the community, designed to enable the open web for the future. On this web, everything can be created from new currencies to new applications to new industries, opening the door to an entirely new future.

1. INTRODUCTION

The richness of the web is increasing day by day with the combined efforts of millions of people who have benefited from “innovation without permission” as content and applications are created without asking anyone. this lack of freedom of data has led to an environment hostile to the interests of its participants. And as we explained in the summary previously, web hosting companies have hindered innovation and greatly monopolized the web.

In the future, we can fix this by using new technologies to re-enable the permissionless innovation of the past in a way, which creates a more open web where users are free and applications are supportive rather than adversarial to their interests.

Decentralization emerged after the global financial crisis in 2008, which created fundamental problems of confidence in the heavily indebted banking system. Then the decentralized financial sector based on Blockchain technology has emerged since 2009.

Decentralized Blockchain technology has made it easy for decentralized digital currencies like Bitcoin to exchange billions of dollars in peer-to-peer transfers for a fraction of the price of a traditional banking system. This technology allows participants in the over $ 50 billion virtual goods economy to track, own and trade in these commodities without permission. It allows real-world goods to cross into the digital domain, with verified ownership and tracking just like that of the digital.

By default, the Internet where freedom of data enables innovation will lead to the development of a new form of software development. On this web, developers can quickly create applications from open state components and boost their efforts by using new business models that are enabled from within the program itself rather than relying on parasitic relationships with their users. This not only accelerates the creation of applications that have a more honest and cooperative relationship with its users, but also allows the emergence of completely new business built on them.

To enable these new applications and the open web, it needs the appropriate infrastructure. The new web platform cannot be controlled by a single entity and its use is not limited due to insufficient scalability. It should be decentralized in design like the web itself and supported by a community of distributors widely so that the value they store cannot be monitored, modified or removed without permission from the users who store this value on their behalf.

A new decentralization technology (Blockchain), which has facilitated decentralized digital currencies like Bitcoin, has made billions of dollars in peer-to-peer transfers at a fraction of the price of the traditional banking system. This technology allows participants in the $ 50 billion + virtual goods economy to track, own and trade in these goods without permission. It allows real-world goods to cross into the digital domain, with verified ownership and tracking just like that of the digital.

Although the cost of storing data or performing a calculation on the Ethereum blockchain is thousands and millions of times higher than the cost of performing the same functionality on Amazon Web Services. A developer can always create a “central” app or even a central currency for a fraction of the cost of doing the same on a decentralized platform because a decentralized platform, by definition, will have many iterations in its operations and storage.

Bitcoin can be thought of as the first, very basic, version of this global community-run cloud, though it is primarily used only to store and move the Bitcoin digital currency.

Ethereum is the second and slightly more sophisticated version, which expanded the basic principles of Bitcoin to create a more general computing and storage platform, though it is a raw technology, which hasn’t achieved meaningful mainstream adoption.

1.1 WHY IS IT IMPORTANT TO PAY THE EXTRA COST TO SUPPORT DECENTRALIZATION?

Because some elements of value, for example bits representing digital currency ownership, personal identity, or asset notes, are very sensitive. While in the central system, the following players can change the value of any credits they come into direct contact with:

1. The developer who controls the release or update of the application’s code

2. The platform where the data is stored

3. The servers which run the application’s code

Even if none of these players intend to operate with bad faith, the actions of governments, police forces and hackers can easily turn their hands against their users and censor, modify or steal the balances they are supposed to protect.

A typical user will trust a typical centralized application, despite its potential vulnerabilities, with everyday data and computation. Typically, only banks and governments are trusted sufficiently to maintain custody of the most sensitive information — balances of wealth and identity. But these entities are also subject to the very human forces of hubris, corruption and theft.

Especially after the 2008 global financial crisis, which demonstrated the fundamental problems of confidence in a highly indebted banking system. And governments around the

world apply significant capital controls to citizens during times of crisis. After these examples, it has become a truism that hackers now own most or all of your sensitive data.

These decentralized applications operate on a more complex infrastructure than today’s web but they have access to an instantaneous and global pool of currency, value and information that today’s web, where data is stored in the silos of individual corporations, cannot provide.

1.2 THE CHALLENGES OF CREATING A DECENTRALIZED CLOUD

A community-run system like this has very different challenges from centralized “cloud” infrastructure, which is running by a single entity or group of known entities. For example:

1. It must be both inclusive to anyone and secure from manipulation or capture.

2. Participants must be fairly compensated for their work while avoiding creating incentives for negligent or malicious behavior.

3. It must be both game theoretically secure so good actors find the right equilibrium and resistant to manipulation so bad actors are actively prevented from negatively affecting the system.

2. NEAR

NEAR is a global community-run computing and storage cloud which is organized to be permissionless and which is economically incentivized to create a strong and decentralized data layer for the new web.

Essentially, it is a platform for running applications which have access to a shared — and secure — pool of money, identity and data which is owned by their users. More technically, it combines the features of partition-resistant networking, serverless compute and distributed storage into a new kind of platform.

NEAR is a community-managed, decentralized cloud storage and computing platform, designed to enable the open web in the future. It uses the same core technology for Bitcoin and Blockchain. On this web, everything can be created from new currencies to new applications to new industries, opening the door to an entirely new future.

NEAR is a decentralized community-run cloud computing and storage platform, which is designed to enable the open web of the future. On this web, everything from new currencies to new applications to new industries can be created, opening the door to a brand new future.

NEAR is a scalable computing and storage platform with the potential to change how systems are designed, how applications are built and how the web itself works.

It is a complex technology allow developers and entrepreneurs to easily and sustainably build applications which reap the benefits of decentralization and participate in the Open Web while minimizing the associated costs for end users.

NEAR creates the only community-managed cloud that is strong enough to power the future of the open web, as NEAR is designed from the ground up to deliver intuitive experiences to

end users, expand capacity across millions of devices, and provide developers with new and sustainable business models for their applications.

The NEAR Platform uses a token — also called “NEAR”. This token allows the users of these cloud resources, regardless of where they are in the world, to fairly compensate the providers of the services and to ensure that these participants operate in good faith.

2.1 WHY NEAR?

Through focus, we find that Platforms based on blockchain technologies like Bitcoin and Ethereum have made great progress and enriched the world with thousands of innovative applications spanning from games to decentralized financing.

However, these original networks and none of the networks that followed were not able to bridge the gap towards mainstream adoption of the applications created above them and do not provide this type of standard that fully supports the web.

This is a result of two key factors:

1. System design

2. Organization design

System design is relevant because the technical architecture of other platforms creates substantial problems with both usability and scalability which have made adoption nearly impossible by any but the most technical innovators. End-users experience 97–99% dropoff rates when using applications and developers find the process of creating and maintaining their applications endlessly frustrating.

Fixing these problems requires substantial and complex changes to current protocol architectures, something which existing organizations haven’t proven capable of implementing. Instead, they create multi-year backlogs of specification design and implementation, which result in their technology falling further and further behind.

NEAR’s platform and organization are architected specifically to solve the above-mentioned problems. The technical design is fanatically focused on creating the world’s most usable and scalable decentralized platform so global-scale applications can achieve real adoption. The organization and governance structure are designed to rapidly ship and continuously evolve the protocol so it will never become obsolete.

2.1.1 Features, which address these problems:

1. USABILITY FIRST

The most important problem that needs to be addressed is how to allow developers to create useful applications that users can use easily and that will capture the sustainable value of these developers.

2. End-User Usability

Developers will only build applications, which their end users can actually use. NEAR’s “progressive security” model allows developers to create experiences for their users which more closely resemble familiar web experiences by delaying onboarding, removing the need for user to learn “blockchain” concepts and limiting the number of permission-asking interactions the user must have to use the application.

1. Simple Onboarding: NEAR allows developers to take actions on behalf of their users, which allows them to onboard users without requiring these users to provide a wallet or interact with tokens immediately upon reaching an application. Because accounts keep track of application-specific keys, user accounts can also be used for the kind of “Single Sign On” (SSO) functionality that users are familiar with from the traditional web (eg “Login with Facebook/Google/Github/etc”).

2. Easy Subscriptions: Contract-based accounts allow for easy creation of subscriptions and custom permissioning for particular applications.

3. Familiar Usage Styles: The NEAR economic model allows developers to pay for usage on behalf of their users in order to hide the costs of infrastructure in a way that is in line with familiar web usage paradigms.

4. Predictable Pricing: NEAR prices transactions on the platform in simple terms, which allow end-users to experience predictable pricing and less cognitive load when using the platform.

2.1.2 Design principles and development NEAR’s platform

1. Usability: Applications deployed to the platform should be seamless to use for end users and seamless to create for developers. Wherever possible, the underlying technology itself should fade to the background or be hidden completely from end users. Wherever possible, developers should use familiar languages and patterns during the development process. Basic applications should be intuitive and simple to create while applications that are more robust should still be secure.

2. Scalability: The platform should scale with no upper limit as long as there is economic justification for doing so in order to support enterprise-grade, globally used applications.

3. Sustainable Decentralization: The platform should encourage significant decentralization in both the short term and the long term in order to properly secure the value it hosts. The platform — and community — should be widely and permissionlessly inclusive and actively encourage decentralization and participation. To maintain sustainability, both technological and community governance mechanisms should allow for practical iteration while avoiding capture by any single parties in the end.

4. Simplicity: The design of each of the system’s components should be as simple as possible in order to achieve their primary purpose. Optimize for simplicity, pragmatism and ease of understanding above theoretical perfection.

2.2 HOW NEAR WORKS?

NEAR’s platform provides a community-operated cloud infrastructure for deploying and running decentralized applications. It combines the features of a decentralized database with others of a serverless compute platform. The token, which allows this platform to run also, enables applications built on top of it to interact with each other in new ways. Together, these features allow developers to create censorship resistant back-ends for applications that deal with high stakes data like money, identity, assets, and open-state components, which interact seamlessly with each other. These application back-ends and components are called “smart contracts,” though we will often refer to these all as simply “applications” here.

The infrastructure, which makes up this cloud, is created from a potentially infinite number of “nodes” run by individuals around the world who offer portions of their CPU and hard drive space — whether on their laptops or more professionally deployed servers. Developers write smart contracts and deploy them to this cloud as if they were deploying to a single server, which is a process that feels very similar to how applications are deployed to existing centralized clouds.

Once the developer has deployed an application, called a “smart contract”, and marked it unchangeable (“immutable”), the application will now run for as long as at least a handful of members of the NEAR community continue to exist. When end users interact with that deployed application, they will generally do so through a familiar web or mobile interface just like any one of a million apps today.

In the central cloud hosted by some companies today like: Amazon or Google, developers pay for their apps every month based on the amount of usage needed, for example based on the number of requests created by users visiting their webpages. The NEAR platform similarly requires that either users or developers provide compensation for their usage to the community operators of this infrastructure. Like today’s cloud infrastructure, NEAR prices usage based on easy to understand metrics that aren’t heavily influenced by factors like system congestion. Such factors make it very complicated for developers on alternative blockchain-based systems today.

In the centralized cloud, the controlling corporation makes decisions unilaterally. NEAR community-run cloud is decentralized so updates must ultimately be accepted by a sufficient quorum of the network participants. Updates about its future are generated from the community and subject to an inclusive governance process, which balances efficiency and security.

In order to ensure that the operators of nodes — who are anonymous and potentially even malicious — run the code with good behavior, they participate in a staking process called “Proof of Stake”. In this process, they willingly put a portion of value at risk as a sort of deposit, which they will forfeit if it is proven that they have operated improperly.

2.2.1 Elements of the NEAR’s Platform

The NEAR platform is made up of many separate elements. Some of these are native to the platform itself while others are used in conjunction with or on top of it.

1. THE NEAR TOKEN

NEAR token is the fundamental native asset of the NEAR ecosystem and its functionality is enabled for all accounts. Each token is a unique digital asset similar to Ether, which can be used to:

a) Pay the system for processing transactions and storing data.

b) Run a validating node as part of the network by participating in the staking process.

c) Help determine how network resources are allocated and where its future technical direction will go by participating in governance processes.

The NEAR token enables the economic coordination of all participants who operate the network plus it enables new behaviors among the applications which are built on top of that network.

2. OTHER DIGITAL ASSETS

The platform is designed to easily store unique digital assets, which may include, but aren’t limited to:

• Other Tokens: Tokens bridged from other chains (“wrapped”) or created atop the NEAR Platform can be easily stored and moved using the underlying platform. This allows many kinds of tokens to be used atop the platform to pay for goods and services. “Stablecoins,” specific kinds of token which are designed to match the price of another asset (like the US Dollar), are particularly useful for transacting on the network in this way.
• Unique Digital Assets: Similar to tokens, digital assets (sometimes called “Non Fungible Tokens” (NFTs) ranging from in-game collectibles to representations of real-world asset ownership can be stored and moved using the platform.

3. THE NEAR PLATFORM

The core platform, which is made up of the cloud of community-operated nodes, is the most basic piece of infrastructure provided. Developers can permissionlessly deploy smart contracts to this cloud and users can permissionlessly use the applications they power. Applications, which could range from consumer-facing games to digital currencies, can store their state (data) securely on the platform. This is conceptually similar to the Ethereum platform.

Operations that require an account, network use, or storage at the top of the platform require payment to the platform in the form of transaction fees that the platform then distributes to its community from the authentication contract. These operations could include creating new accounts, publishing new contracts, implementing code by contract and storing or modifying data by contract.

As long as the rules of the protocol are followed, any independent developer can write software, which interfaces with it (for example, by submitting transactions, creating accounts or even running a new node client) without asking for anyone’s permission first.

4. THE NEAR DEVELOPMENT SUITE

Set of tools and reference implementations created to facilitate its use by those developers and end users who prefer them. These tools include:

• NEAR SDKs: NEAR platform supports (Rust and AssemblyScript) languages to write smart contracts. To provide a great experience for developers, NEAR has a full SDK, which includes standard data structures, examples and testing tools for these two languages.
• Gitpod for NEAR: NEAR uses existing technology Gitpod to create zero time onboarding experience for developers. Gitpod provides an online “Integrated Development Environment” (IDE), which NEAR customized to allow developers to easily write, test and deploy smart contracts from a web browser.
• NEAR Wallet: A wallet is a basic place for developers and end users to store the assets they need to use the network. NEAR Wallet is a reference implementation that is intended to work seamlessly with the progressive security model that lets application developers design more effective user experiences. It will eventually include built-in functionality to easily enable participation by holders in staking and governance processes on the network.
• NEAR Explorer: To aid with both debugging of contracts and the understanding of network performance, Explorer presents information from the blockchain in an easily digestible web-based format.
• NEAR Command Line Tools: The NEAR team provides a set of straightforward command line tools to allow developers to easily create, test and deploy applications from their local environments.

All of these tools are being created in an open-source manner so they can be modified or deployed by anyone.

3. ECONOMIC

Primarily economic forces drive the ecosystem, which makes up the NEAR platform. This economy creates the incentives, which allow participants permissionlessly organize to drive the platform’s key functions while creating strong disincentives for undesirable, irresponsible or malicious behavior. In order for the platform to be effective, these incentives need to exist both in the short term and in the long term.

The NEAR platform is a market among participants interested in two aspects:

• On the supply side, certification contract operators and other core infrastructure must be motivated to provide these services that make up the community cloud.
• On the demand side, platform developers and end-users who pay for their use need to be able to do so in a simple, clear and consistent way that helps them.

Further, economic forces can also be applied to support the ecosystem as a whole. They can be used at a micro level to create new business models by directly compensating the developers who create its most useful applications. They can also be used at a macro level by coordinating the efforts of a broader set of ecosystem participants who participate in everything from education to governance.

3.1 NEAR ECONOMY DESIGN PRINCIPLES

NEAR’s overall system design principles are used to inform its economic design according to the following interpretations:

1. Usability: End users and developers should have predictable and consistent pricing for their usage of the network. Users should never lose data forever.

2. Scalability: The platform should scale at economically justified thresholds.

3. Simplicity: The design of each of the system’s components should be as simple as possible in order to achieve their primary purpose.

4. Sustainable Decentralization: The barrier for participation in the platform as a validating node should be set as low as possible in order to bring a wide range of participants. Over time, their participation should not drive wealth and control into the hands of a small number. Individual transactions made far in the future must be at least as secure as those made today in order to safeguard the value they modify.

3.2 ECONOMIC OVERVIEW

The NEAR economy is optimized to provide developers and end users with the easiest possible experience while still providing proper incentives for network security and ecosystem development.

Summary of the key ideas that drive the system:

• Thresholded Proof of Stake: Validating node operators provide scarce and valuable compute resources to the network. In order to ensure that the computations they run are correct, they are required to “stake” NEAR tokens, which guarantee their results. If these results are found to be inaccurate, the staker loses their tokens. This is a fundamental mechanism for securing the network. The threshold for participating in the system is set algorithmically at the lowest level possible to allow for the broadest possible participation of validating nodes in a given “epoch” period (½ of a day).
• Epoch Rewards: Node operators are paid for their service a fixed percentage of total supply as a “security” fee of roughly 4.5% annualized. This rate targets sufficient participation levels among stakers in order to secure the network while balancing with other usage of NEAR token in the ecosystem.
• Protocol treasury: In addition to validators, protocol treasury received a 0.5% of total supply annually to continuously re-invest into ecosystem development.
• Transaction Costs: Usage of the network consumes two separate kinds of resources — instantaneous and long term. Instantaneous costs are generated by every transaction because each transaction requires the usage of both the network itself and some of its computation resources. These are priced together as a mostly-predictable cost per transaction, which is paid in NEAR tokens.
• Storage Costs: Storage is a long term cost because storing data represents an ongoing burden to the nodes of the network. Storage costs are covered by maintaining minimum balance of NEAR tokens on the account or contract. This provides indirect mechanism of payment via inflation to validators for maintaining contract and account state on their nodes.
• Inflation: Inflation is determined as combination of payouts to validators and protocol treasury minus the collected transaction fees and few other NEAR burning mechanics (like name auction). Overall the maximum inflation is 5%, which can go down over time as network gets more usage and more transactions fees are burned. It’s possible that inflation becomes negative (total supply decreases) if there is enough fees burned.
• Scaling Thresholds: In a network, which scales its capacity relative to the amount of usage it receives, the thresholds, which drive the network to bring on additional capacity are economic in nature.
• Security Thresholds: Some thresholds, which provide for good behavior among participants are set using economic incentives. For example, “Fishermen” (described separately).

4. NEAR’S TECHNOLOGY

NEAR’s community-operated cloud uses a novel consensus algorithm and a scalable sharding architecture to achieve its high-level design goals.

The key elements of NEAR’s technology are:

• Sharding: The system is designed to scale horizontally and near-infinitely by distributing computation across multiple parallelized shards.
• Consensus: Consensus is achieved across all of the nodes which make up the network operators across all of the shards using the new Nightshade algorithm.
• Staking Selection and Game Theory: To participate in the validation process, stakers are selected using a secure randomized process which optimally distributes seats across parties and provides incentives for them to operate with good behavior.
• Randomness: NEAR’s randomness approach is unbiasable, unpredictable and can tolerate up to 1/3 of malicious actors before liveness is affected and 2/3 of malicious actors before any one can actually influence its output.

4.1 TECHNOLOGY DESIGN PRINCIPLES

NEAR’s overall system design principles are used to inform its technical design according to the following interpretations:

• Usability: End users should be burdened with the lowest possible security obligations for a given type of interaction. Developers should be able to easily build, test and deploy contracts in familiar languages and should be able to provide their end-users with experiences close to today’s web .
• Scalability: The platform should scale infinitely as its capacity is used.
• Simplicity: The design of each of the system’s components should be as simple as possible in order to achieve their primary purpose .
• Sustainable Decentralization: The barrier for participation in the platform as a validating node should be set as low as possible in order to bring a wide range of participants. Individual transactions made far in the future must be at least as secure as those made today in order to safeguard the value they modify.

4.2 TECHNICAL SUMMARY

NEAR focuses on providing solutions to the two core problems of today’s blockchains — Usability and Scalability.

Usability for end users is achieved through offering a progressive security model for wallet interactions and by giving developers more opportunities to craft experiences which closely resemble the web today. Flexible and programmable key management implemented on the protocol level provides these. Things like contract-based accounts, meta transactions, atomic account transfers, accounts with funds that are locked for specific usage and other account programmability and restriction use cases are easily enabled with this.

Usability for developers is provided by setting up the protocol to provide for browser-based debugging, familiar programming languages (like AssemblyScript and Rust) and contract usage rebates.

Scalability is provided by sharding the chain into a potentially unlimited number of subchains, each of which operates in parallel.

4.2.1 PERFORMANCE CHARACTERISTICS AND TRADEOFFS

One commonly referenced trilemma states that a system cannot achieve scalability, decentralization and security at the same time. NEAR’s sharding and validator selection approaches provide significant scalability and decentralization while mitigating tradeoffs in security that would normally occur with such improvements.

Another classic trilemma is posed by the CAP theorem, which states that a system can only achieve 2 of Consistency, Availability (aka “liveness”) and Partition Tolerance. Given that

partition tolerance cannot be sacrificed in this case, the tradeoff is really between consistency and availability.

In blockchain-based systems, an illustrative example is what happens if the network splits into two parts for a week. A *consistent* system will completely shut down one (or both) of the halves until the network is restored so that the two parts do not become inconsistent. An *available* system (like Bitcoin) will continue to run both halves of the network independently and, when they are restored to unity, the operations of one-half will be wiped out in favor of the operations of the other.

NEAR currently favors availability at a system level but individual users can choose to not accept blocks without >50% signature thresholds as a way of locally requiring consistency as well.

The performance of the system will be highly dependent on the types of transactions, which are processed, and the actual hardware, which is supporting it. For simple financial transactions, per-shard throughput could range from 400–2000 transactions per second.

4.3 SHARDING

Current approaches to scalability typically fall into two categories:

1. Vertical Scaling: Achieved by improving the performance of the existing hardware of a system. In the case of blockchain-based systems, it typically means running a network containing fewer nodes that each require *better* hardware. This creates an initial improvement in throughput while limiting future improvements to roughly the rate of increase in performance of computing hardware (often considered to be “Moore’s Law”). This leaves the network without the ability to scale at a rate commensurate with its adoption.

2. Horizontal Scaling: Achieved by adding *more* hardware to a system. In the case of blockchains, this is typically done by ensuring that an increase in the number of nodes participating in the network improves the performance of that network by a commensurate amount, for example by parallelizing computation across multiple “shards”.

NEAR uses a sharding approach to provide scalability horizontally, which allows it to scale capacity alongside increases in demand.

4.3.1 CROSS-CHAIN AND CROSS-SHARD COMMUNICATION

One of the biggest difficulties with any form of cross-chain communication, whether this occurs within the shards of a single chain or across multiple chains, is determining that an incoming transaction from another chain is valid. There are 3 approaches for validating cross-chain transactions:

1. Dual Validation: Have the validators for the receiving chain also validate on the sending chain. This is used by Quarkchain. It has the downside that validators do not scale well in this approach.

2. Trust the Transaction: Assume that if a transaction has been received, it must be valid. In Cosmos, for example, a transaction that is copied to the main hub is considered irreversible. They keep track of the total number of tokens in each economy so you cannot create new ones but you could theoretically create invalid transfers between parties (eg steal tokens from other parties).

3. Beacon Chain w/ Rollback: A beacon chain verifies the state transitions all of the other chains using a small subset of validators and, if a problem is detected, all chains are rolled back. To achieve atomicity, this reversion should happen, though it should happen only rarely and should be immediately detected.

NEAR focuses on the 3rd approach. With the assumption that an adaptive adversary cannot corrupt the validators of a shard within a day, validators of each shard can be rotated daily to help add a layer of security. But presumably it is possible (if very difficult) for an adaptive adversary to corrupt a shard’s validators within a given day.

To help offset this, other protocols use a smaller committee which rotates far more rapidly (eg every few minutes) and validates across shards. In order for this smaller committee to perform their validations without having to download the entire state of each shard (which cannot be done in this timeframe), they receive only that portion of the state which was actually affected. But it is difficult to send the state with each change — a single transaction might affect 100mb of state at a time.

This is where the Nightshade sharding approach comes in.

4.4 NIGHTSHADE

Nightshade modifies the typical sharding abstraction and assumes that all of the shards combine together to produce a single block. This block is produced with a regular cadence regardless of whether each individual shard has produced its “chunk” for that specific block height. So every chunk for each shard will either be present or not.

There must be a fork choice rule to decide on the proper fork. This is still under development but will most likely resemble LMD Ghost. It will include the weight of how many validator attestations have been received for a given chunk and block.

There is a single validator assigned to produce each block. This validator must assemble the chunks which are provided to it during that block’s time period into the period’s block. The assignment of this validator will rotate through the existing validator set (eg 100 validators). This leader does not accept transactions, only chunks.

For each individual shard and period, a single validator is assigned to produce its chunk. If that validator is not present, the shard will stall for that period. Each shard has its own smaller pool of validators which is pulled from the main pool. The shard leader position rotates among this smaller pool (eg 4 validators) in the same way that the overall block leader is selected. Thus, if a single validator is absent and the shard chunk stalls for one period, the next validator will likely be present to continue the chain’s operation in the following period.

4.5 HIDDEN VALIDATORS

In order to provide additional security, NEAR uses Hidden Validators. These are a smaller committee for each shard (on average 100 validators) who verify each chunk. Rather than having this assignment be on the blockchain and thus publicly visible to all participants, the validators themselves figure out their assignment individually by drawing a set of shard ids from a Verifiable Random Function (VRF).

This way each individual validator is aware of which shards they must verify but, to corrupt them, an adversary must bribe a large percentage of total validators across all shards to reveal their masks.

Further, the number of hidden validators assigned to a particular block is randomly determined as well. This prevents an adversary from knowing exactly how many hidden validators they even need to corrupt in the first place in order to successfully pull off an attack. This prevents attacks where an adversary broadcasts their intent and waits for the fishermen to come to them (revealing which shards they are validating).

Due to the nature of the verification, any single hidden validator can present a proof that the chunk is invalid, a so-called “fraud proof”.

The selection of the smaller per-shard committee is done for every epoch (½ day) from the same pool as the block and chunk producers, which is the total set of nodes which staked. For example, if there are 100 seats per shard and 100 shards, there are a total of 10,000 seats. 100 of them will be allocated to be the chunk producers and the rest will be hidden validators.

4.6 FISHERMEN

In addition to the hidden validators who are assigned to provide security for each shard, any other node operator can participate permissionlessly as a so-called “fisherman.” This third-party node can provide the same fraud proof as a hidden validator and thus they too can kick off the process of slashing and rollback.

This means that, even if an adversary successfully corrupted the entire hidden validator pool, they have no guarantee that their efforts will not be discovered by one of these independent fishermen and are thus highly disincentivized.

4.7 PREVENTING LAZY VALIDATORS

One potential problem with validators is that they can be “lazy”. After every block, a validator must receive the new chunk, download the new state and run validation on that block. They could, however, choose to do nothing unless they see another validator submitting a fraud proof and only then do they actually validate the latest block and try to submit a proof of their own. Thus a chain can end up paying validators but receive no meaningful work from them.

This is mitigated by making validators to first commit to their decision (if chunk is valid or invalid) and then reveal what they committed. This creates an incentive to do proper work because they have value at stake and will be slashed if they miss an invalid chunk.

4.8 PREVENTING DATA HOARDING

Another problem can occur if a chunk is corrupted (by corrupting its small set of chunk producers) and the chunk producers refuse to provide sufficient data to hidden validators so they are unable to validate or make a fraud proof.

This is solved by requiring the chunk producer to send out an “erasure coded” chunk to other chunk producers in other shards. This code allows these other producers to reconstruct the chunk from just 16% of the parties and hidden validators who can validate it. If the (presumably corrupt) chunk producer did not provide this for their chunk, then no other chunk producer will attest or build on top of the chain, which they don’t have parts for. The “fork choice rule” will select the chain that actually has at least 50% of parties having parts.

To prevent bad behavior, a single random part of this erasure code is deliberately made to be a “land mine” (invalid). At any time, if a validator is shown to have attested to a block, which contains the land mine (which is easily proven), they will be slashed. Thus for each period there is also a small chance that a bad actor gets slashed so it highly disincentivizes bad behavior.

4.9 RANDOMNESS

Randomness in the blockchain needs to have the following properties:

1. Unbiasable

2. Unpredictable

3. Liveness, i.e. tolerates actors going offline or malicious actors

There are a few potential approaches:

1. RANDAO — unpredictable but biasable. Liveness depends on the underlying consensus protocol;

2. RANDAO+VDF — unpredictable, unbiasable, has liveness. But in practice it is hard to use it and be ASICS-resistant at the same time;

3. Threshold Signatures — unpredictable, unbiasable, has liveness. But requires a complicated mechanism to generate private keys in a particular fashion. It is an active area of research at the moment.

4. RandShare — unpredictable, unbiasable, has liveness. But requires O(n³) network communication messages, which is a lot, where n is the number of participants. And also becomes biasable with more than 1/3 malicioius participants which is a low threshold.

NEAR’s approach is unpredictable, unbiasable and has liveness. Unlike Randshare, it tolerates up to 2/3 malicious participants before it becomes biasable. Unlike Threshold Signatures, it is simple. Unlike RANDAO+VDF, it can not be attacked with ASICs.

5. GOVERNANCE

Governance defines how the protocol is updated (“Technical Governance”) and how its resources are allocated (“Resource Governance”). Technical governance generally includes fixing bugs, updating system parameters and rolling out larger scale changes in the fundamental technology of the protocol. Resource governance generally includes the allocation of grant funding from community-initiated sources (like the allocation provided to the foundation).

Technical governance is particularly complex because of the required coordination between potentially thousands of independent node operators around the world. Each of those nodes must go through the upgrade process in order to participate in the newest version of the network. Any who do not may end up (attempting to start) running a separate chain. Thus it is important that the upgrade process is smooth and that the nodes it affects buy into the decisions that have been made.

Many protocols perform the decision-making aspect of governance “off chain”, meaning it occurs in text channels, in-person and via phone calls where key stakeholders or their representatives decide on the best course of action.

Other protocols lean heavily on “on chain” governance, where decisions are explicitly made by the holders of the protocol’s key resources (eg tokens) via an online voting mechanism. This provides explicit clarity around decision making and rollouts but suffers from the need to very clearly specify each case.

5.1 NEAR GOVERNANCE DESIGN PRINCIPLES

Ease of use: Governance processes must be clear and understandable. Mechanisms for active participation and voting (where available) should be simple and direct. Governance must be effective and efficient in order to reach decisions quickly and implement them effectively. The stakeholder community must have a sufficient voice to support the legitimacy of decisions and not to leave or branch off the platform.

• Scalability: Governance should expand as the platform and complexity of the platform itself grow, as stakeholder diversity increases and participation increases.
• Simplicity: The strongest processes tend to be the simplest, so good governance must avoid excessive engineering processes and recognize that human-to-human communication is the simplest approach.
• Sustainable decentralization: Governance must allow participation from the full range of stakeholders on the platform, but it must be flexible in addressing any of these over time.

It is important for the governance design to balance efficiency and flexibility. Decisions must be taken and implemented efficiently if the technical platform wants to continue to develop sufficiently to provide the best value to its stakeholders, but this platform must ensure that it cannot be captured over time by a specific group of stakeholders.

5.2 GOVERNANCE SUMMARY

NEAR’s governance is designed to provide for efficient improvement of the protocol while allowing the community sufficient voice and oversight in order to ensure the protocol maintains its independence. The long term goals are to combine community led innovation with effective decision making and execution and to receive proper representation from each of the key stakeholder roles in the network.

For example, the NEAR community initially includes token holders, validators, application developers, protocol developers, community leaders and more. Each of these stakeholders has a different set of views, opinions and inputs to various key issues.

Having proper representation means that decisions will require deliberation and discussion, which can slow down the necessary evolution of the protocol if left unchecked. To maintain a bias for efficient execution, a highly qualified entity is needed to maintain the Reference Implementation of core protocol code. This maintainer, who is called the Reference Maintainer, should be selected and overseen by the community.

Initially, governance activities are coordinated by the NEAR Foundation, an independent nonprofit entity whose mission is well-aligned with improving the long term usefulness of the ecosystem. These activities include maintaining oversight over the Reference Maintainer, supporting the build up of the governance coordination tools, certain token distributions and laying the groundwork for community operated governance.

5.3 TECHNICAL GOVERNANCE

As a decentralized network, no single entity can ever force changes to the full NEAR network. The nodes who are running the network must individually accept any changes made to the reference code base by its core contributors.

It is still important to understand the core process, which is used to push changes to the reference code base because these are most likely to represent the will of the community and thus receive acceptance by the network nodes, which form part of that community.

NEAR’s governance defines a Reference Maintainer, which is an entity responsible for technical upgrades to the NEAR network. This entity has been selected to maintain the Reference Implementation and continue to suggest improvements on the specification. All major releases are guarded with community discussion and a veto process (a 2 week challenge period), while smaller bug fixes can be rolled out fast and delivered to node operators.

Initially, the Maintainer is selected by the Foundation Board and serves until the board votes to replace them. Over time, oversight of the Maintainer will be performed through a community-representing election process.

6. NEAR COMMUNITY

Anyone in the world can contribute to NEAR’s technical, content or community efforts through the Contributor program.

1-Build the Future Web

Contribute in whatever way fits your skills — whether translations, bug fixes, event hosting, creative design or more.

2-Have an Impact

Shape the project and steer its future while learning and building a platform for yourself in the blockchain space.

3-Get Rewarded

Each contribution is rewarded with tokens so you can build a stake in the result of your efforts.

7. NEAR TEAM

The teams that make up the NEAR Collective have built successful companies and implemented some of the only real-world sharded systems at scale.

8. NEAR ROADMAP

MainNet Roadmap and Timeline

We are going to be releasing NEAR’s MainNet in stages. Each stage is identified by the restrictions that it has and each stage has different goals. It’s important as we open up the network more and more to test at every stage and provide flexibility to address issues early in the process.

NEAR ROADMAP

8.1 MAINNET POA

Expected time to launch: April 22–27, 2020

This is NEAR Protocol running in Proof of Authority mode, with the NEAR Foundation operating the initial set of nodes. Most importantly, the state of the network will be maintained going forward.

The goal for this stage is to distribute initial tokens to contributors and get the initial set of validators onboard. At this point, only the NEAR Foundation is able to transfer tokens and will be using lock-up account contracts when allocating tokens to first users. In other words, most token transfers are restricted.

Developers who are ready to deploy their applications to MainNet can apply to the NEAR Foundation and get an account to deploy their application.

In parallel, we are still running our TestNet with various validators to test out all the corner cases of validation. As soon as both NEAR Collective and validators are satisfied with the stability of running on TestNet, we will transition into the next phase.

8.2 MAINNET (RESTRICTED)

Expected time to launch: June — August, 2020

Given that transfers are disabled for most accounts and the lockup contracts doesn’t allow to stake directly (only via delegation), the initial validator set is determined by a whitelist of validators to delegate to. As soon as the initial set of validators is determined from TestNet and shows their MainNet infra running, NEAR Foundation will stop staking and pass the staking to these validators.

There are a few goals for this phase:

• Test that MainNet is operating correctly with a decentralized set of validators and continue code and security reviews
• Initial applications which can work without transfer restrictions can start launching
• NEAR Foundation will be continuously distributing tokens to the value-add community.

This stage is completed when the community decides that the network is secure and decentralized enough. A contract to vote on lifting transfers will be used by the community to identify such time. Validators will be casting votes, as they have locked capital in the network and can not “double vote”. Delegators can cast their votes via staking contract as proxy, overruling validators vote proportionally to their stake.

When ⅔ of total stake and at least 35% of total supply vote for a specific block number, it’s considered decided by the community and transfers will unlock in two weeks from that block number.

8.3 MAINNET (COMMUNITY GOVERNED)

Expected time: decided by the community

At this point, the network is fully operational and doesn’t have any restrictions. Validators and delegators are now responsible for continued operation of the network and deciding to upgrade.

In parallel with the general community gaining confidence and voting to unlock transfers, NEAR Collective will continue to work on ensuring the quality and security of the network, and has released a target to finalize a number of post-flight tasks.

8.4 POST MAINNET

One of the critical parts for any project is to continue moving forward and evolve over time. To make sure NEAR does this, we follow a development process that establishes a fixed launch timeline. We have setup three networks to provide us with a testbed:

• devnet — nightly released network from the master branch that provides a place to stress test and do initial testing of code added that day, additionally run nightly test suit.
• betanet — released weekly, accepts external validators and application developers who want to live on the bleeding edge.
• testnet — released every 4 weeks and is a stable version, this is the same TestNet we have been running since April 2019, and is the best place to do acceptance testing for one’s applications.

Validators on MainNet will be voting as a community on accepting new stable releases. We expect that the timing on this would usually depend on the magnitude of changes that came in that release.

We are not going to launch MainNet with all the features we have in mind. Many things end up being outside of our “MainNet v1” scope to make sure we can deliver quality products. Some of these include:

• Upgradability without hard forking (link to discussion and epic tracking issues related)
• Unbiasable randomness (design, implementation)
• Safes: safe locks for operations with assets across contracts and shards (design)
• Enable challenges to switch to selfish majority assumption (epic)
• Parallelize Runtime (epic)
• Rework Storage (epic)
• Dynamic resharding (epic)

 And more features and improvements coming from community and developers

After the “Community Governed” phase is done, we are going to set up the next release of MainNet v2 with a set of tasks to complete in subsequent 4 weeks.

9. REFERENCES

 https://near.org/blog/mainnet-roadmap/

 https://near.org/team/

 https://near.org/technology/