Before we get to the verification flow, here is a short overview of how we develop, deploy, and upgrade our contracts.

Contracts: Develop, Deploy, Upgrade

We use the 0xweb stack of libraries: 0xweb, 0xweb-hardhat, dequanto

This gives us the following workflow:

1. Write Solidity contracts as usual.
2. Run npx hardhat compile --watch. The 0xweb plugin compiles the Solidity files and generates, for each contract, a TypeScript or JavaScript client class. Each generated class encapsulates the blockchain interaction layer and exposes a clean interface.

The --watch flag triggers recompilation as soon as you save modified Solidity files.

Each generated class is linked to the Hardhat artifacts, which are used by the deployment factory:

import { MyFooContract } from '@0xc/hardhat/MyFooContract/MyFooContract';
import { Deployments } from 'dequanto/contracts/deploy/Deployments';
import { Web3ClientFactory } from 'dequanto/clients/Web3ClientFactory';

let client = await Web3ClientFactory.getAsync('eth');
let deployments = new Deployments(client, deployer, {
    directory: './deployments/',
    whenBytecodeChanged: 'redeploy'
});

let { contract: foo } = await deployments.ensure(MyFooContract, {
    arguments: someConstructorArgs
});

// Read method
let someValue = await foo.getSomeValue();
// Write method
let tx = await foo.$receipt().setSomeValue(deployer, 123);

The ensure method checks whether the contract is already deployed on the selected network and whether the bytecode matches. If the bytecode is unchanged, it returns a ready-to-use instance; otherwise, it deploys the contract. After deployment it also verifies the contract on the blockchain explorer (for Ethereum this is Etherscan).

This works well for stateless contracts. For stateful contracts you usually need upgradeable proxies, so that the storage stays intact while the implementation changes. The deployment factory supports TransparentUpgradeableProxy through the ensureWithProxy method:

let { contract: foo } = await deployments.ensureWithProxy(MyFooContract, {
    arguments: someConstructorArgs,
    initialize: someInitializerArgs
});

Here, the factory compares the deployed bytecode with the current build. If there are changes, it redeploys the implementation contract and submits an upgradeToAndCall transaction.

Important: the ProxyAdmin should not be your deployer account. It should be a Timelock, or at least a Multisig. A safe setup looks like this:

MyFooContractProxy ← Timelock ← Multisig

This ensures that upgrade transactions cannot be executed immediately. Dequanto supports many account types (Timelock, Multisig, ERC-4337), but we will not go into those details here.

The key point is: you now have a pending upgrade transaction that will replace the implementation of your proxy. Before approving it, you must compare the old and new implementations. Below are the steps we use.

Implementation Comparison Workflow

1. Check out the exact commit used for deployment

Every deployment must correspond to a specific git commit. If you don't have it locally, fetch it with minimal history:

git clone --no-checkout my-repo
cd my-repo
git fetch --depth 1 origin <commit-hash>
git checkout <commit-hash>

You now have the exact sources that were used for deployment.

2. Download verified sources for the old and new implementations

npm i 0xweb -g
0xweb i <old-implementation-address> --chain eth --name myFooV1
0xweb i <new-implementation-address> --chain eth --name myFooV2

After this step, you have three sets of sources:

• repository sources from the referenced commit,
• verified sources for the old implementation,
• verified sources for the new implementation.

3. Compare the new implementation against the old one

git diff --no-index ./0xc/eth/myFooV1/myFooV1/contracts/ ./0xc/eth/myFooV2/myFooV2/contracts/

This diff shows exactly what changed between versions.

4. Compare repository sources against the new implementation

git diff --no-index --diff-filter=DM ./0xc/eth/myFooV2/myFooV2/contracts/ ./my-repo/contracts/

We use the DM filter to show only deleted and modified files, because the repository will often include extra files that are not part of the verified sources.

The goal here is simple: the output should be empty. If there is no diff, then the verified implementation matches your codebase.

These two comparisons are essential. Each party involved in the upgrade process should review them before signing and executing the upgrade transaction through the multisig.

§ pro et contra

• 🚀 Zero setup time

It is already up and running. Simply define your data model as a contract and deploy it.

• ✨ Zero maintenance

Once your data is uploaded, it remains accessible as long as the blockchain operates. I can assume that it will be far longer than your other hosting subscription.

• 💯 100% Read uptime; close to 100% Write uptime

The separation of reading and writing processes in blockchain ensures 100% uptime for read operations, especially when leveraging multiple RPC providers for redundancy.

• 🛡️ Secure

Blockchains inherently provide a higher level of security than conventional hosting solutions. Data exploits are possible only if vulnerabilities exist in your data model’s logic.

• 📖 Open Data

Unless encrypted, your data remains open, accessible, and verifiable by anyone, promoting transparency.

• 🖧 DNS-Free

Domain names are unnecessary for this type of backend. Instead, a list of decentralized node providers can be used, allowing client libraries to select the most efficient option for end-users.

• 🤝 Trust

Thanks to the features above, blockchain-based backends inherently build user trust by ensuring data security and 24/7 availability, even if project maintenance and development stops.

• 🧩 Interepolibility with 3rd party data models

You can integrate other data models stored on the blockchain, or other projects can build upon your data model.

• ⚙️ Extensibility

Users can leverage numerous third-party projects to monitor or automate actions, significantly expanding the possibilities of your data model.

• 📜 History and time-travel

The data can be accessed from any point in the past.

• 📡 Events and event-streams

Load historical custom events or use WebSockets to listen for real-time incoming events, enabling dynamic application responses.

• 👤 Built-in user identity

The “wallet” concept enables users to authenticate themselves by signing messages, providing seamless and decentralized user identification.

• 📝 Grant users to modify or extend data

Users can modify or extend data in your storage based on the permissions you define. Importantly, the costs of these modifications are borne by the users themselves. By selecting a low-cost blockchain, these fees can remain negligible, often amounting to only a few cents per transaction.

• 🌐 Huge and continuously evolving ecosystem

• numerous solidity modules you can use to enhance your data model's functionality.

• many open-source and free services, as well as those offering free plans, are available. It is a common practice within the blockchain community to provide free plans that are often sufficient for production needs.

§ contra

• 💾 Storage is expensive 😢

Though it follows a true pay-as-you-go model, you pay just for the SLOTs you store in. Each SLOT has 32 bytes, it costs 20000 GAS to write new data or 5000 GAS to update the data. Let's take Polygon as an example, with a 30-gwei GAS price and a $0.60 POL price.

\(20000GAS 30gwei = 0.008 POL $0.60 = $0.00032\)

This is a lot, so the “Floppy Disk“ emoji represents the storage amounts in the best way, which means it is best suited for smaller datasets if you pay on your own. However, a unique advantage is that users can bear the costs of their own storage and actions, a feature not found in other technologies. While this approach might hinder the mass adoption of your app, it is widely accepted within the blockchain community.

• 🧮 Computation is limited 😢

Blockchain data models support functions for interacting with the data, but their computational capabilities have constraints. These limitations depend on the RPC nodes you use for read actions and the strict GAS limits imposed on write actions (transactions). While basic operations, loops, and deeper call stacks are typically manageable, the blockchain is unsuited for heavy computational workloads.

That said, given the relatively small data sizes typically involved, the existing limits are usually sufficient for most use cases.

§ punctum neutrum

• 🧬 Data structure, Solidity language, SDKs

If you're new to blockchain development, you may have heard that it’s complicated and hard to get started with. However, this is not true. Blockchain development uses familiar concepts, semantics, and syntax, making it easier to learn than it might seem.

§ Demo: Application Version Repository

https://github.com/0xweb-org/examples-backend

For this article, let’s create an application version manager contract. Imagine you have a desktop application that requires a backend to check for new versions and retrieve the download link whenever a new version is published. Below is the final contract, demonstrating most of the key concepts:

import { Ownable } from "@openzeppelin/contracts/access/Ownable.sol";

struct Package {
    uint version;
    uint timestamp;
    string url;
    bytes32 sha256;
}

contract AppVersionManager is Ownable {

    // Events that are emitted on data updates
    event NewApplicationInfo();
    event NewPackage(uint version, uint timestamp);

    // Custom error, when title for the application is empty
    error TitleIsEmpty();

    // Some application information
    string public title;
    // @TODO: add further application related properties if required

    // Latest package
    Package public package;

    // Track all versions and their packages
    mapping (uint => Package) public packages;

    // List of all previous versions
    uint[] public versions;

    constructor () Ownable(msg.sender) {
    }

    function updateInfo(string calldata newTitle) external onlyOwner {
        if (bytes(newTitle).length == 0) {
            revert TitleIsEmpty();
        }
        title = newTitle;
        emit NewApplicationInfo();
    }

    function updatePackage(Package calldata newPackage) external onlyOwner {
        require(newPackage.version > package.version, "Newer package already published");

        packages[package.version] = package;
        package = newPackage;
        versions.push(package.version);
        emit NewPackage(package.version, block.timestamp);
    }

    function findPackageAtTimestamp (uint timestamp) external view returns (Package memory) {
        if (package.timestamp <= timestamp) {
            return package;
        }
        // the countdown loop to find the latest package for the timestamp
        int i = int(versions.length);
        while (--i > -1) {
            Package memory pkg = packages[versions[uint(i)]];
            if (pkg.timestamp <= timestamp) {
                return pkg;
            }
        }
        revert("No package found");
    }

    function getPackage (uint version) external view returns (Package memory) {
        if (version == package.version) {
            return package;
        }
        return packages[version];
    }
}

Every developer can read and understand this code with minimal effort. If you’re familiar with TypeScript, most of the concepts here will already make sense. To make it even clearer, I’ve created an equivalent TypeScript example: AppVersionManager.ts 🔗.

In simple terms, a contract in Solidity can be thought of as a stateful class instance. The concepts of properties, methods, types, and inheritance are already well-known in object-oriented programming. The main concept to explain here is the onlyOwner modifier (similar to a decorator in TypeScript).

Every blockchain account is essentially a pair of private and public keys. The account's ID, known as the address, is derived from the public key. When a transaction is executed, the sender’s address is passed as msg.sender. Using this, we can store your address in the constructor (during contract deployment). Later, the onlyOwner modifier ensures that only you, as the contract owner, can execute the updateInfo and updatePackage functions. If someone else attempts these actions, the transaction will be reverted. The onlyOwner modifier is provided by the Ownable contract, which is part of the widely-used OpenZeppelin library. This library includes many other useful contracts to streamline blockchain development.

Another important topic to discuss is the concept of Proxies, which split storage and implementation into two separate contracts. Contract implementations in Solidity are immutable, meaning you cannot add new functions or properties after deployment. To work around this, you can deploy a “Proxy” contract. The Proxy handles storage and contains only one fallback function, which delegates calls to the implementation contract while maintaining the storage context of the Proxy.

This concept may sound complex, but it’s similar to how this works in JavaScript. Here’s a quick analogy to help clarify:

const foo = new Proxy({ bar: 'Lorem' }, {
    get (obj, prop) {
        return fooImplementation[prop].bind(obj)
    },
});
const fooImplementation = { logValue () { console.log('Bar value:', this.bar) } }

foo.logValue();

The proxy contract holds a reference to the implementation contract. If you want to add new functions, you simply deploy a new implementation contract and update the proxy to reference this new contract, forwarding function calls to the updated instance. It’s a straightforward process, but there’s an edge case to consider: constructors.

When deploying an implementation contract, its constructor operates within the storage of the implementation contract itself. This means that setters like title = "Hello World" will not modify the proxy's storage. To address this, we use the initializer function concept:

1. Deploy the implementation contract with a initialize function.

2. Deploy the proxy contract, passing the address of the implementation contract in its constructor. This setup allows the initialize method to be called in the context of the proxy contract.

As a result, updating the title property, for example, will correctly update it in the proxy’s storage.

Here’s an upgraded implementation version of our AppVersionManager: AppVersionManagerUpgradeable.sol.

The proxy contract itself is quite universal and independent of the implementation. Several well-known standards for proxies are available in the OpenZeppelin library.

With the knowledge of these concepts and the examples above, you’re ready to develop smart contracts for your business cases.

§ Deployment

1. Select the blockchain

First, we need to select the blockchain where we want to deploy our contract. For this example, I’ve chosen Polygon. It offers low transaction costs, has been around for a long time, and has consistently performed well. Its stable and efficient infrastructure, combined with a Total Value Locked (TVL) of $0.9 billion, makes it a reliable choice. Deploying your contracts to public blockchains means coexisting with financial institutions. The TVL metric reflects the trust these institutions place in the blockchain’s reliability.

Moreover, if conditions change, you can always redeploy the contract to another blockchain in the future.

2. Deploy

The demo project also serves as the CI test repository, so all the commands can be found here: https://github.com/0xweb-org/examples-backend/blob/master/deploy-cli.sh

# Install 0xweb library from NPM into the prject folder
npm i 0xweb
# Install required dependencies to compile/deploy *.sol files
npx 0xweb init --hardhat --openzeppelin
# Create or import the account. Private key will be encrypted with pin AND machine key.
npx 0xweb accounts new --name foo --pin test --login
# Save the private key securly and ensure the account has some POL tokens
# Deploy. The foo account is selected as default.
npx 0xweb deploy ./contracts/AppVersionManager.sol --chain polygon --pin test
# Set title
npx 0xweb c write AppVersionManager updateInfo --newTitle MySuperApp --pin test
# Set latest package information
npx 0xweb c write AppVersionManager updatePackage --arg0 'load(./data/package.json)' --pin test

With just a few commands, you’ve deployed the contract and updated the data. That’s it for the backend—it's now up and running “forever” without requiring any further actions from your side. The costs for this deployment, at a GAS price of 70 gwei and a POL price of $0.51, would be:

You spend just 4 cents to set up a decentralized, secure, and long-running service with no maintenance required.

§ Query

To query your contract data, you’ll need RPC node providers. Dozens of free providers are available at https://chainlist.org. You can choose multiple providers, and a good Web3 library can utilize a round-robin strategy at runtime to select the most efficient one for your end users. With 0xweb, the generated TypeScript or JavaScript classes not only select the best endpoints but also abstract away all blockchain communication. The clients contain high-level methods for fetching data, making the process seamless and efficient.

# The deploy command also generates the class, but manual install is also possible
npx 0xweb i 0x<address> --name AppVersionManager --chain polygon

import { AppVersionManager } from './0xc/polygon/AppVersionManager/AppVersionManager'

const manager = new AppVersionManager();
console.log(`Title`, await manager.title());
console.log(`Package`, await manager.package());

For other programming languages, there are numerous libraries available to simplify querying the blockchain. After deployment, you’ll have the contract address and ABI (interface).

Alternatively, you can launch a middleware server to query contract data using 0xweb.

npx 0xweb server start --port 3000
curl http://localhost:3000/api/c/read/AppVersionManager/package?chain=polygon

One advantage is that you don’t need to include any libraries in your application - raw HTTP requests. However, this approach relies on an additional server that you’ll need to manage. It’s often better to query the blockchain directly using 0xweb-generated classes or other blockchain libraries available.

§ Summary 🏁

This article showcased how blockchains can be both simple and powerful, offering unique advantages compared to traditional hosting solutions.

In the next article, I plan to explore decentralized BLOB storage networks such as Greenfield and Arweave, highlighting their features and benefits.

If you have any suggestions or ideas for additional features to include in the 0xweb library, feel free to share them in the comments or reach out directly at tnbts@0xweb.org.

In the previous article, we looked into how the EVM storage is arranged and how we can access all the contract's data relatively easily when the contract is validated or we have the source code. To simplify it even further, there is the storage handler in the dequanto library and the 0xweb CLI tool. I've published the storage demo repository at github/examples-storage, which is used for the 0xweb CI pipeline, and I'll go through the example in this article.

We start with the contract's class generation:

$ npm i 0xweb -g
$ 0xweb init --hardhat
$ 0xweb i 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 --name USDC --chain eth

This fetches the contract sources from etherscan, detects it as a proxy, and fetches the implementation at 0xa2327a938febf5fec13bacfb16ae10ecbc4cbdcf. After the source code and the ABI are present 0xweb generates the TypeScript class for the contract: USDC.ts - that, beyond the read and write methods of the contract, also has the storage handler to read the state variables and to overwrite state variables data in the hardhat environment.

EVM Storage Forking

To play around with the storage we can fork the Ethereum Mainnet storage with a hardhat development server. Hardhat launches the local chain that "inherits" the complete state. Earlier I thought, this process was very time and resource-consuming, but in reality, it is a simple RPC proxy. When you call some method of the contract, for instance balanceOf of the USDC contract and the local server doesn't have any code yet for the address, it fetches this from the forked RPC endpoint with getCode, when the code gets executed and it reads the balances[account] the slot location is not yet present locally, so it will fetch the data with single getStorageAt to get the slot and resume contract execution. Hardhat allows you to overwrite the local storage. That is why forking is a great feature of developing, testing, and performing on-chain research.

Once left to mention - forking per default gets the latest block number and all proxy requests are made with the block number, but you can specify any block number you want to fork the remote chain from and perform operations as if the contract has all the state of that point of time. For this reason, when forking you should have an archive node. But, if you use latest block number for forking and don't have access to the archive node, your requests will still work for some time (latest is fetched for a specific number), depending on how quickly your node clears the state - for geth this is 128 blocks, which is around 27 minutes.

Run demo actions

I use atma and atma-utest to execute typescript actions. The actions you can see in actions/usdc.act.ts

[1] Create accounts

atma act actions/usdc.act.ts -q "create-accounts"

Prepares two accounts, which we will use to fund, overwrite and transfer balances.

[2] Overwrite balance for account "foo"

atma act actions/usdc.act.ts -q "set-balance"

The generated TypeScript class has the storage field with autogenerated getters for all state variables

Foreword: Storage readers

let usdc = new USDC();
let amount: bigint = await usdc.storage.totalSupply_();

This reads the internal variable directly from the storage of the USDC contract:

contract FiatTokenV1 is AbstractFiatTokenV1, Ownable, Pausable, Blacklistable {
    // ...
    mapping(address => uint256) internal balances;
    mapping(address => mapping(address => uint256)) internal allowed;
    uint256 internal totalSupply_ = 0;
    // ...
}

Though there is an external method totalSupply() in the contract, but it depends on the contract developers if they decide to present getters manually or set variable visibility to public. That's why the contract may not have those getters, but with the storage handler, we can read all state variables of the contract.

So, after we generate the storage handler class, it contains methods to get all contract variables. The methods are strongly typed - with arguments and correct return types, but there is a more generic method storage.$get(accessorPath) to get variable chains like the JSON's paths, e.g. myContract.storage.$get('foo.bar.qux')

Consider such a contract:

contract Foo {
    struct Position { 
        address owner;
        uint amount
    }
    Position[] positions;
}

We can read for example the address of the owner on the index 1 with:

let foo = new Foo();
let owner = await foo.storage.$get('positions[1].owner');
// with generated method we would get the complete position struct
let position = await foo.storage.positions(1);
// OR with $get
let position = await foo.storage.$get('positions[1]')
// the position variable has the interface of IPosition
interface IPosition {
    owner: TAddress
    amount: bigint 
}

Setters

Writing to the storage works only in the development environment and it can be done only via the "$set" method. Now, back to our action set-balance — we must set the new amount to the storage slot of the user balance.

let amount = 50n * 10n**6n // 50 USDC
let address = foo.address;
await usdc.storage.$set(`balances["${address}"]`, amount);

[3] Transfer balance

That's it - now we can use the newly updated balance, and transfer the tokens to the account "bar".

let tx = await usdc.transfer(foo as ChainAccount, bar.address, fooBalance);
let receipt = await tx.wait();

Other hardhat development methods:

• hardhat_setBalance - we can add any amount of ETH to the account, for example before submitting the test transaction.

• hardhat_setCode - we can change the contract's code

• hardhat_impersonateAccount - send transactions from any address

🏁 Happy coding

Even though the data is publicly readable, Projects can still decide how transparent they are ready to go, and this level of transparency is visible to everyone. How much data are they ready to store on the blockchain? Do they publish the ABIs, the source code? Do they encrypt data?

As you may know, when the ABI is published, you can query the contract's data and logs. But developers may decide to make some state variables private, so the ABI for fetching its data won't be created. This is usually done, to hide the state variables from other contracts, but for you, as an external observer, the data is open. It's just more difficult to locate its storage slot to read the value with getStorageAt, in comparison to reading the Data provided via the Application Binary Interface.

Here I briefly explain how the contract's storage works in EVM and in Part II show the tool, which can generate the TypeScript classes with data-getter methods for all state variables based on the source code. In the dev or research environment, you can even use the data-setters to overwrite the data.

Storage

The storage is divided into 32-byte blocks (the slots). You can think about it, as a sparse Array of blocks. The maximum size of that array is — 2²⁵⁶ elements. A sparse — means an array with "holes", or gaps in the sequence of their indices. For example in JavaScript:

let foo = [];
foo[5] = 'A';
foo[999000] = 'B';

Though the length of the Array foo is 999001, it contains only two elements in the memory, so the indices are virtual memory pointers. I think this is important to know, to understand later how those indices (the storage locations) are calculated. But before we continue with the storage, let us go into Data-Types, and in particular their Memory Sizes.

Types

1. Simple fixed-size value types

2. Complex fixed-size types

3. Variable-size types

Storage Layout

From the table, it is easy to recognize, that there is a huge range of storage locations(indexes) and each storage slot has a fixed size - of 32bytes.

Storage Locations

Simple fixed-size value types

So far we know already the required number of bytes to store different variable types, and we know how the storage is divided into slots. Now let's see where the EVM stores the data for state variables and we will start from the simple contract:

contract FooContract {
    address foo;
    bool isActive;
    uint256 amount;
}

From the example, we can notice the order of declared state variables: foo is 0, isActive is 1, amound is 2. If there were more variables the order would proceed. The compiler takes this order into consideration. And so, our common sense would say each variable would occupy the slot with the index of the variable. And that's it, but with one exception: why would the "isActive" occupy the entire slot 1 when the data has only 1 byte, and in the previous slot, we have some space left (the address takes only 20bytes out of 32bytes)? Right, we can store the boolean in "slot 0", then the "amount" goes to the next slot 1 . Another question you may have is, why the data of the "amount" variable moves to the next slot if we still have some space left (11bytes) "slot 0": address(20bytes)+boolean(1byte)=21 But, as the uint256 takes the complete 32bytes we would need to split the data along 2 slots, and as the EVM reads the data per WORD basis (one slot at once) it is much better to save the base types in one slot, therefore amount variable occupies one complete slot 1 and the rule is very simple:

If we can store the complete value in the current slot, we do so, otherwise go-to the next slot and save there.

Complex fixed-size types

Here is an example with complex fixed-size state variables

contract FooContract {
    struct User {
        address foo;
        bool isActive;
        uint256 amount;
    }
    User user;
    User[3] users;
}

I explicitly took the same simple types for the User struct to show, that the logic of data locations is also the same:

• when assigning slot numbers to a struct type, EVM assigns slot numbers to underlying slot variables.

• when assigning slot numbers to a fixed array type, EVM assigns slot numbers to elements.

So you can think about those types, as just the logical "grouping" of data, which is represented in storage with the same locations, as without the "grouping".

contract FooContract {
    // User
    address foo;
    bool isActive;
    uint256 amount;

    // User[3]
    // User0
    address User0_foo;
    bool User0is_Active;
    uint256 User0_amount;
    // User1
    address User1_foo;
    bool User1_isActive;
    uint256 User1_amount;
    // and so on ...
}

Here we already know, how to locate and pack the data from the previous step.

Variable-size types

Again, let's start with the example:

contract FooContract {
    address[] users;
    bool isActive;
}

The previous logic won't work, as we don't know the number of elements in users array, so we can't place the data before isActive boolean, while the location of isActive would be dependent on this dynamic length of users.

The isActive the location must follow the previous rules: a) it is the 2nd variable b) we can't store it in the previous slot - which means, the isActive must stay in the slot 1 .

What do we save in 0 slot?

In case of array we must save at least the length of the array, otherwise users.length won't be possible. So length of the array goes directly into the slot 0.

Where do we save the array items?

And also we want that they are nonfragmented*, have deterministic locations*, and not collide* with other contract variables.

• Not fragmented — we select the location in storage of the element 0 and any other goes directly after that location:

  locationOf(users[n]) === locationOf(users[0]) + n

• Deterministic locations — if we want to access e.g. the item users[14] we should be able to calculate the storage location without any reads from the storage.

• No collisions — there should be no collisions with other contracts variable states, nor they should be mixed or grouped. So the item locations must be isolated.

EVM solves all requirements above with a simple trick: they use keccak256 function to hash the slot number of a dynamically sized state variable. The hash is the uint256 number, which is used as a new starting position in the storage for the items. You can think about hashing as a "jump" in memory. So as we know, the users variable has the initial slot 0, the location of the users[14] will be:

const stateVariableSlotNr = 0;
let hash = keccak256(encodePacked({
    value: stateVariableSlotNr, 
    type: 'uint256'
}));
let jump = BigInt(hash); // new slot number
let users14Location = jump + 14n;
//hex: 0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e571
//bigint:18569430475105882587588266137607568536673111973893317399460219858819262702961

The such approach solves all requirements above:

• Not fragmented: After the "jump" we can store the dynamic array in the same way as we do it with fixed-size arrays - the elements follow one after another by incrementing the slot number.

• Deterministic locations: From the source code, the compiler knows the initial slot number of the variable, that's why we can quickly calculate the "jump" with keccak256

• Isolated: You can see in the example above how huge the "jump" number could be, with no chance for other variables to end up in the same location.

How do we save dynamic arrays of complex types?

Let's go one step further and look into complex array types, not as simple address[] as in the previous example.

contract FooContract {
    struct User {
        address foo;
        bool isActive;
        uint256 amount;
    }
    User[] users;
}

It turns out - everything works the same way, for example, to get the location of users[14].amount

• From the source code, the compiler knows the slot index of users = 0

• From the source code, the compiler knows that each element occupies 2 slots Remember packing?

  0 slot = foo(20bytes) + isActive(1byte)

  1 slot = amount(32bytes)

const stateVariableSlotNr = 0;
const slotsPerElement = 2n;
const amountPropertySlotNr = 1n;
const index = 14n;
let hash = keccak256(encodePacked({
    value: stateVariableSlotNr, 
    type: 'uint256'
}));
let jump = BigInt(hash); // new slot number
let users14AmountLocation = jump 
    + slotsPerElement * index 
    + amountPropertySlotNr;

What about nested dynamic arrays?

I hope from the previous examples, it won't be complicated for you to solve this task on your own - users[5].balances[8] :

contract FooContract {
    struct User {
        address foo;
        uint256[] balances;
    }
    User[] users;
}

• users is the slot 0

• users: the location of the first element in the array starts at

  UsersCursor = BigInt(keccak256(bytes32(0)))

• User consists of 2 slots:

  0 slot = foo-address

  1 slot = balances-array-length

• users[5]: the location of the 5th item is

  Item5Cursor = UsersCursor + 5 * 2

• users[5].balances: the location of the first item in the balances array starts at: Item5BalancesCursor = BigInt(keccak256(bytes32( Item5Cursor + 1 ))

• users[5].balances[14]: the location of the item is

  SlotNumber = Item5BalancesCursor + 14

Mappings

Another dynamically sized type is mapping. Unfortunately, it is not possible to arrange the storage for "Mapping" in "Not fragmented" and "Deterministic locations" ways at once. The Array indices are sorted, but the Mapping keys are not, so here we select determinism over defragmentation, but we apply here the similar logic of "jumps" for every key, as for the 0th item of the array. You remember - the location of the 0th item was: keccak256(bytes32(ARRAY_SLOT_NR)) With mapping items, we will also use the "MAPPING_SLOT_NR" and combine it with the key for a "jump" to every value.

let jump = BigInt(keccak256(encodePacked({
    value: bytes32(key),
    type: 'bytes32'
}, {
    value: bytes32(slotNr),
    type: 'bytes32'
})));

In ARRAY_SLOT_NR we have the array length, but in MAPPING_SLOT_NR we have nothing as there is no mapping.length, that is why we don't need any extra information in the mapping slot itself.

It is not possible to iterate over every mapping key, as there is no place in storage where the keys are stored. You must know the key to be able to locate its storage slot. That is why, it is important for a well-implemented contract, to emit the "Log", every time a mapping item is created. We know all the holders of an ERC20 token, not by reading the balances mapping, but by iterating the Transfer logs.

Complex mapping values

contract FooContract {
    struct User {
        address foo;
        bool isActive;
        uint256 amount;
    }
    mapping(address => User) users;
}

Everything works here exactly the same way: after we have calculated the location of the element by address key and the users stat variable slot number - at that index the foo and isActive are stored. The amount will be at the next slot (+ 1)

Strings and Bytes

string and bytes are not the same as byte[], bytes32 or bytes32[]

• bytesX are fixed-sized entities in the storage, like uintX, address, etc.

• bytesX[] are arrays of fixed-sized entities, like uintX[], address[], etc.

• bytes and string are the dynamic-sized buffers in storage, that have slightly different storage layouts, but are very similar to arrays.

In the array's slot we store the number of items in the array. For bytes and strings, we store the size (bytes count) of the data.

EVM also applies a nice trick to pack short strings/byte buffers. If the size of the data is less then 32 bytes, we could store the size and the data in the same slot: 31 bytes for data, and 1 byte for the size number. Otherwise, the data is split into slots, which are stored in the same manner as arrays. For example, when we have the 100 bytes data, it will occupy Math.ceil(100/32) slots.

Inheritance and multiple inheritance

This is important for the slot order of the state variables. Let's look directly into the example:

contract Bar {
    uint256 bar;
}
contract Foo {
    uint256 foo;
}
contract Qux is Foo, Bar {
    uint256 qux;
}

The inheritance chain defines the order of storage variables, so the qux won't occupy the 0 slot. From the example:

0 slot = foo
1 slot = bar
2 slot = qux

For a more complex example, like the deeper inheritance chain - the rule of thumb stays the same:

The slot number of the first variable in a contract is the incremented slot number of the last variable from the previous contract in the inheritance chain.

Conclusion 🏁

By understanding the core concepts of storage variable order, basic type sizes, packing, and the "jump"s you have a clear view of EVMs storage.

You also see that the calculation of the slot locations for arbitrary contracts and variables is complicated when done manually, that's why we embedded the storage reader functionality into the TypeScript contract class generator - 0xweb. And we'll look into it in Part II.

Externally owned accounts (EOAs)

These are the addresses and private keys we all use to communicate with the blockchain. There is no other way to do this, but the problem is - people directly communicate with the dApps - tokens, NFTs, stacking, DAO voting, and so on.

As the result, all assets are attached to that account, and it turns into a wallet, which holds all your funds. Though there are many ways you can try to secure your private 🔑KEY, all of them mainly go to checkpoints on how to secure your Operating System. Even hardware wallets can't protect your funds if your system is compromised and the malicious transactions are subscribed with the KEY. And I'm not talking about browser extensions, such as MetaMask, where in the worst case the attacker can get the seed phrase.

We had already the case when a person in our team had downloaded the malware from a phishing website, and his funds from the accounts, he used in MetaMask, were gone. He, a technically aware person, got a victim of the Oski Stealer.

But there is a much better way to secure the assets - to have multiple and rotatable 🔑KEYs with all funds on multi-sig.

Multi-signature accounts (Contracts)

A multi-signature account is a contract - living on-chain and being a proxy between your KEYs and the dApps.

And as result, this is the wallet that holds all your assets. The multi-sig is like your personal guard, which can bypass only valid transactions, there are some rules you can specify - the number of required signatures, and spending limits.

🔑 Multiple Keys

For example, you can create a 2-of-2 scenario - you generate one private key on your machine, and the second on your mobile phone, and you set both addresses as owners into the multi-sig, after that, to create a transaction you need both signatures to add to it so that the multi-sig passes the transaction through. And this is the 2FA for transactions:

1. Create a transaction on your machine with the first signature.
2. Review and sign the transaction on your mobile.

You still have to protect your devices, so that the KEYs stay safe, but this drastically increases the security. This mitigates even phishing attacks, as you 👀 double-check the transactions.

🔃 Rotatable

While your assets are owned by your multi-sig account and your EOAs are just the KEYs to that account, you can create new KEYs and replace them in the multi-sig at any time. So if you have any doubts you could be compromised - replace the affected address. Or you can do this on a regular basis - once half a year - change the KEYs.

✳️ A noncustodial solution, and no vendor locks

I saw some beliefs, that Gnosis Safe Multisig is a custodial wallet, for those people — see the section about Gnosis Safe below.

🔥 Edge cases and the caveats

There are some situations when you can't use multi-sigs, but all of them are just limited by the dApp implementations.

• Message signing. EOAs can sign messages with their associated private keys, but currently, contracts cant. There is a solution for this — eip-1271, but it should be implemented and supported by the dApp

• tx.origin. Not so common as the previous one, but some contracts explicitly check msg.sender == tx.origin, which means they restrict forwarded transactions and require the caller to be an EOA, not the contract. I've met this only once by Alpaca Finance.

• slightly higher ⛽GAS consumption

• No easy-to-use solution yet, like the MetaMask browser extension. It is possible to connect via WalletConnect, though it is also limited to allowed dApps.

Gnosis Safe

I use Gnosis Safe multi-signature contracts, as the most established solutions in this area. As I've said previously, this is not a custodial multi-sig, to understand this, you should know how it works. It consists of 2 parts:

1. Multisig Contract. Once you've created this on-chain, this contract belongs to you. You are the owner, and only you have access to it. Gnosis shares its implementation with you so that you could create the contract.

2. Off-chain Transport. Gnosis web app and Gnosis Safe mobile applications are just the clients for your multi-sig contract, but you can create multi-sig transactions and collect signatures without Gnosis Safe infrastructure.

As an example, you can send your desired transaction data to participants via e-mail as a JSON file and collect the signatures back. The transport doesn't need to be secure, a man-in-the-middle can only see what transaction you want to send, but can't manipulate it. Also, no private keys are disclosed.

0xWeb

Here I will show, how I use the 2FA for my transactions. 0xWeb is a CLI wallet with the ability to install validated contracts from Etherscan and other EVM blockchain explorers — it generates TypeScript client classes that can be used via CLI or API. But to keep it simple, for an example, I will create a simple transfer transaction.

Create or Add EOA/Safe accounts

1. 📱Create a "FooMobile" account in Gnosis Safe App. Go to "Settings 🠖 Owner Keys 🠖 ADD 🠖 Create new owner Key"
2. 🖥️ Create a "FooDesktop" account in 0xWeb CLI.

    $ 0xweb accounts new --name FooDesktop --pin <pin>
   

3. 🖥️ Finally create the multi-sig. We add to members the address of the account we've created in GnosisSafe

   $ 0xweb safe new --name safe/foo --owner FooDesktop --members 0x<...> --chain eth
   

4. 📱Add the Multisigs Address to Gnosis Safe App Go to "Menu🠖 Add Safe 🠖 Select Chain 🠖 Enter the address"

You see, I've created the keys on different devices, with different vendors, and none of the will store both private keys.

Perform arbitrary transactions, for example, transfers

Assume we sent some ETH to the safe, and later want to transfer some from the safe to another address.

$ 0xweb transfer 0.04 ETH --from safe/foo --to 0x<...> --pin <pin> --chain eth

After I run this command, I receive the push notification with the Tx Proposal on my mobile, I review and confirm the transaction, something like this:

0xweb tool waits until the Gnosis API Service returns the required signature and submits the Tx to the blockchain.

Backend

Not only for personal use but also for backends we use the multi-sig approach. There is an additional validator instance. We store all assets on multi-sigs, and for example, when a user wants to withdraw assets, in our main backend we validate the intent, create a tx proposal, and send these to the validator backend, that one validates the intent with the tx once again, and sends its confirmation back to the main server, which sends the Tx to the blockchain.

Summary

I'm still wondering, why the EOA wallets are still popular to store funds on when MultiSigs are much more secure. I hope browser wallets, like MetaMask, will add the MultiSig support, so we can connect to dApps with the multi-sig account, and before the transaction is submitted, we have to confirm it with our mobile app.

Stay safe. 🔒

In the first part, we'll go through the commands required to install and configure things, then you'll get a script to automate this, to be able to perform all these steps with a single command.

The stack

1. Nginx

• as we may want to have multiple Node.js applications and domains on a single server
• static file server
• SSL certificates

2. Certbot

• we will create letsencrypt certificates

3. PM2

• for managing Node.js processes and clusters. Optionally, you can enable monitoring with keymetrics

4. Local Git

• to push the code, build and restart the apps

5. Node.js

• n module helps to manage the nodejs versions

Install

apt update
apt install nginx nodejs npm certbot -y
npm install -g n
# update nodejs to the latest
n stable
node -v
npm -v
npm i pm2 -g
# start pm2 on OS start
pm2 startup

Git configuration

# the server directory
mkdir -p /var/www/foo
# the git repository, we will push to
git init --bare /var/www/foo.git

Then we have to create the post-receive hook - the script, which will be executed each time we push the changes to the git server. In the script we will a) copy the code to the server directory b) handle git submodules c) run build scripts d) restart the nodejs process:

cat <<EOT >> /var/www/foo.git/hooks/post-receive
#!/bin/sh
git --work-tree=/var/www/foo --git-dir=/var/www/foo.git checkout -f
cd /var/www/foo
# optionally, restore submodules
git --git-dir=/var/www/foo.git --work-tree=. submodule init
git --git-dir=/var/www/foo.git --work-tree=. submodule update
npm i
npm run build
# this will also restart foo, if already running
pm2 start foo
EOT

# allow to execute
chmod +x /var/www/foo.git/hooks/post-receive

Nginx configuration

This one we will split into two steps:

1. create SSL certificates for the domain
2. create the reverse proxy to the server's port

1. SSL

# to remove any default websites
rm /etc/nginx/sites-available/default

# copy-paste simple server for the letsencrypt challange
cat <<EOT >> /etc/nginx/sites-available/default
server {
    listen       80;
    server_name  foo.bar;
    root /var/www/foo;
}
EOT

# start a new server
service nginx restart

# will generate the certificates
certbot certonly --webroot -w /var/www/foo -d foo.bar --email team@foo.bar --agree-tos --no-eff-email

# now create the final server
rm /etc/nginx/sites-available/default

cat <<EOT >> /etc/nginx/sites-available/default
server {
    listen       80;
    listen       443 ssl;
    server_name  foo.bar;

    ssl_certificate  /etc/letsencrypt/live/foo.bar/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/foo.bar/privkey.pem;

    if ($scheme = http) {
        return 301 https://$server_name$request_uri;
    }
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}
EOT

service nginx restart

Project

1. Add ecosystem.config.js file to configure the pm2

module.exports = {
    "apps": [
        {
            "name": "foo",
            "script": "index.js",
            "args": [
                "--SERVER", "--port 3000",
            ],
            "node_args": [
                "--max-old-space-size=12000"
            ],
            "timestamp": "MM-DD HH:mm Z",
            "instances": "max",
            "exec_mode": "cluster",
            "env": {}
        }
    ]
};

1. Add the upstream repository

git init
git remote add prod root@SERVER_IP_DOMAIN:/var/www/foo.git/

You can extend and install any other software you need, which is beneficial by using plain SSH access directly to your Ubuntu instance - there are dozens of other tutorials and configurations.

Execute the commands with a single Node.js script

The commands above are simple copy-paste-enter commands, which means you can submit them one by one, but I'm using ssh2 module to submit commands over the ssh client.

I've also published the ssh2 wrapper class to make the client easier to use - sshly

import { Ssh } from 'sshly'

const script = `
apt update
---
apt install nodejs npm certbot -y
---
npm install -g n
---
n stable
---
npm i pm2 -g
---
pm2 startup
---
mkdir -p /var/www/foo
---
git init --bare /var/www/foo.git
---
cat <<EOT >> /var/www/foo.git/hooks/post-receive
#!/bin/sh
git --work-tree=/var/www/foo --git-dir=/var/www/foo.git checkout -f
cd /var/www/foo
git --git-dir=/var/www/foo.git --work-tree=. submodule init
git --git-dir=/var/www/foo.git --work-tree=. submodule update
npm i
npm run build
pm2 start foo
EOT
---
chmod +x /var/www/foo.git/hooks/post-receive
---
rm /etc/nginx/sites-available/default
---
cat <<EOT >> /etc/nginx/sites-available/default
server {
    listen       80;
    server_name  foo.bar;

    location ^~ /.well-known/acme-challenge/ {
        root /var/www/foo
        allow all;
        default_type "text/plain";
    }
}
EOT
---
service nginx restart
---
certbot certonly --webroot -w /var/www/foo -d foo.bar --email team@foo.bar --agree-tos --no-eff-email
---
rm /etc/nginx/sites-available/default
---
cat <<EOT >> /etc/nginx/sites-available/default
server {
    listen       80;
    listen       443 ssl;
    server_name  foo.bar;

    ssl_certificate  /etc/letsencrypt/live/foo.bar/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/foo.bar/privkey.pem;

    if ($scheme = http) {
        return 301 https://$server_name$request_uri;
    }
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}
EOT
---
service nginx restart
`

const client = new Ssh({
  host: 'IP',
  privateKeyPath: 'path/to/file'
});
const commands = script
  .split('---')
  .filter(x => x.trim().replace(/\r/g, ''))
  .filter(Boolean);
for (let command of commands) {
    await client.exec(command);
}

Publish the app

Push your code to the repository

git add -A
git commit -am "init"
git push -u prod master

Summary

You can create your own setup, using apache2 for example, or separate roles. After all, these are plain *nix commands with Node.js scripting, and it is a huge advantage over using some additional deployment tools or platforms.

Adjust and save the server creation script as a template, so that later you can deploy Node.js servers with ease.

Happy deploying.

To set a proxy in chrome with selenium driver is not a big deal, there are even two options - through the API, the example here or with the command line argument --proxy-server=http://foo.bar. But, after the instance is launched, the reconfiguration of the capabilities has no effect. So here, I'll show a very simple workaround to the issue. We'll go through the theoretical part, and then the small implementation. The examples will contain a couple of Windows-specific things, but for macOS and Linux it would work similar.

1. Use additional local proxy, like squid

Okay, if we can set a proxy for chrome, at least at startup, then our requirement of "changing the proxy" we should move out of the chrome context. For this, we can create an additional local proxy, which supports parent proxies. The ideal option here will be the cross-platform tool - the squid cache(http://www.squid-cache.org/). Installing the package with choco and configuration with vscode is quite simple

$ choco install squid
$ code c:/Squid/etc/squid/squid.conf

Here we need to configure the parent proxy with cache_peer, it supports also the authorization.

cache_peer PARENT_IP parent PARENT_PORT 0  no-query default login=USERNAME:PASSWORD connect-fail-limit=99999999  proxy-only
never_direct allow all

Now, to change the proxy, we need a) to read the configuration, b) to edit those two lines, c) to write the file back, and d) to reconfigure the squid with the command

$ squid -d 0 -k reconfigure

⚠️ The terminal should be launched with admin privileges.

All of these are easily scriptable. We'll go into details later.

2. Keep-Alive sessions.

So now we know, how to change the proxy servers at runtime, but there is a caveat ❗❗— the requests can still go through the old proxy, as the agent can reuse connections and will hold the sockets opened, even in the idle state.

I looked for an API to close all idle connections via selenium, but haven't found it: if anybody has some hints for me, it would be great. But there is another way, we can do it via chrome://net-internals/#sockets, by evaluating the script

chrome.send('closeIdleSockets')

3. Implementation

Part 1.

For modifying the squid config, I will use:

• atma-io for the FileSystem IO
• shellbee for the CLI

import { File } from 'atma-io'
import { run } from 'shellbee';
import { URL } from 'url'

export namespace SquidConfig {
    const SQUID_CONFIG_PATH = 'file://c:/Squid/etc/squid/squid.conf';

    export async function setProxy (httpProxy: string) {
        let { hostname, port, username, password} = new URL(httpProxy);

        let content = await File.readAsync<string>(SQUID_CONFIG_PATH);
        let rgxCheckExists = /^cache_peer .+/m;
        let line = `cache_peer ${hostname} parent ${port} 0  no-query default login=${username}:${password}  connect-fail-limit=99999999  proxy-only`;
        if (rgxCheckExists.test(content)) {
            content = content.replace(rgxCheckExists, line);
        } else {
            content += `${line} \n never_direct allow all`;
        }

        await File.writeAsync(SQUID_CONFIG_PATH, content);
        // re-read the configuration
        let { stderr } = await run(`squid -d 0 -k reconfigure`);
        if (stderr.length > 0) {
            throw new Error(stderr.join('\n'));
        }
    }
}

Part 2.

I'll show how to close the sockets using the selenium-query itself.

import SQuery from 'selenium-query'

export namespace ChromeSockets {
    export async function closeIdleSockets () {
        let $internalsPage: SQuery = await SQuery.load('chrome://net-internals/#sockets');
        await $internalsPage.eval(`chrome.send('closeIdleSockets')`)
    }
}

4. Testing

I use atma-utest module to run the test.

Bootstrap the demo project in some folder:

npm i atma -g
# repair default packages and TS runners
atma init
# install additional packages
npm i selenium-query shellbee

After you've created the files with snippets provided earlier, you can create the test, e.g. ./tests/proxies.spec.ts

import SQuery from 'selenium-query';
import { ChromeSockets } from '../src/ChromeSockets';
import { SquidConfig } from '../src/SquidConfig';
import { URL } from 'url';

// remote proxy servers
const PROXY1 = process.env.PROXY1;
const PROXY2 = process.env.PROXY2;
const IP_PROVIDER = `https://api.ipify.org?format=json`;

UTest({
    async 'should visit both proxies' () {

        await SquidConfig.setProxy(PROXY1);
        // opens new chrome instance
        let { data: server1 } = await SQuery.fetch(IP_PROVIDER, {
            args: [
                '--log-level=3',
                // the local squid proxy
                '--proxy-server=http://localhost:3128'
            ]
        });

        eq_(server1.ip, new URL(PROXY1).hostname);

        await SquidConfig.setProxy(PROXY2);
        await ChromeSockets.closeIdleSockets();
        // reuses chrome instance
        let { data: server2 } = await SQuery.fetch(IP_PROVIDER);

        eq_(server2.ip, new URL(PROXY2).hostname);
    }
});

Finally, run the test

$ atma test tests/proxies.spec.ts

Both servers should be in Environment as strings like, http://USERNAME:PASSWORD@IP:PORT

To position the alot library on a "map" and give you the orientation - I would say, it is somewhere between JavaScript native array methods and Rx, and in no way completely replaces both of them. Alot is highly inspired by LINQ methods from .NET.

TypeScript and installation

The library is implemented in TypeScript and has complete typings support out of the box, but can be used also in the plain js environment.

$ npm i alot

If you, the reader, still do not use TS on your daily dev basis, I would highly encourage you to switch to typescript. In later articles, I will show how I use TypeScript with no configuration and compilation hassle, and use it as it were a JavaScript (as it actually is, but with sweet additions.)

🌱 Laziness

Similar to rx cold observables, the chained methods won't be evaluated until you call subscribe method - the equivalents in alot library: toArray, toArrayAsync, first, firstAsync etc. But let's compare these two examples:

import alot from 'alot'

let arr1 = users
    .filter(user => user.score > 10)
    .map(user => user.name.toUpperCase())
    .slice(0, 5);

let arr2 = alot(users)
    .filter(user => user.score > 10)
    .map(user => user.name.toUpperCase())
    .take(5)
    .toArray()

Though the code here does the same things and looks similar, the flow is different. JavaScript methods do the following:

1. filter all users by some score field
2. map all previously filtered users to their uppercased names and create a new array of strings
3. take(slice) the first 5 usernames.

As you may already have noticed, the caveat here is when the users array has lots of items - the engine goes through all the users to create a new filtered array, then goes over each item to pick the username and create the new array with names.

The example with alot library has the reversed direction flow. When the js engine evaluates filter, map, and take methods, it builds the query that gets executed when we call the toArray() method and the flow is:

1. take 5 elements from the underlying map stream
2. map users to their uppercased names from the underlying filter stream. As only 5 elements were requested it takes and maps also only 5 elements
3. filter users by some score field. As only 5 elements were requested, it processes filtering only until 5 elements are matched

When I'm writing code I still think in forward-flow filter>map>take, but understanding how laziness works under the hood is beneficial.

So, in the best case, if the first 5 users have a score greater than 10, then the js engine visits only those 5 elements for filtering and mapping.

I have to mention, that it works great for this example. If we wouldn't have slice/take methods, then in alot example we would obviously map all filtered users and we wouldn't benefit from the laziness nature of alot lib. That's why I still often use js native array methods, and the alot API allows me quickly switch between different flows.

The laziness brings us to another feature, which was easy to implement - the asynchronous.

⛓️ Asynchronous

All methods in alot library have also there asynchronous variations, which accept async methods. From the previous example, what if we would want to load also user's comments along with uppercased username - for native js methods we have a huge headache, how to accomplish this. I have seen such wrong ❌ solution:

let promises = users
    .filter(user => user.score > 10)
    .map(async user => {
        return {
            username: user.name.toUpperCase(),
            comments: await Api.loadComments(user.id)
        };
    })
    .slice(0, 5);
let arr1 = await Promise.all(promises);

Wrong things about this could be:

• here we load comments for all filtered users, even though we need only 5.
• even, if we would need all users, not only 5, we make anyway all requests parallel - which means if we have an array with 1000 elements - we make 1000 requests simultaneously. (Though browsers and nodejs have max number for open connections, not all requests go immediately to the server, but it could cause timeout errors anyway)

With alot library it would be almost similar and ✅

let arr2 = await alot(users)
    .filter(user => user.score > 10)
    .mapAsync(async user => {
        return {
            username: user.name.toUpperCase(),
            comments: await Api.loadComments(user.id)
        };
    })
    .takeAsync(5)
    .toArrayAsync({ threads: 2})

And the previous issues are solved:

• due to the take method, we map only 5 elements, which means we make only 5 api requests to load the comments.
• alot library can handle the async queue - in this example, we make 2 requests to the server at once (this number is just an example, you can set any number of simultaneous async tasks, default is 4)

So with little code change, we managed to create a lazy asynchronous stream

We can also make our filter async, in case we have to get the data for filtering also async.

let arr2 = await alot(users)
    .filterAsync(user => Api.isActive(user))
    .mapAsync(async user => {
        return {
            username: user.name.toUpperCase(),
            comments: await Api.loadComments(user.id)
        };
    })
    .takeAsync(5)
    .toArrayAsync({ threads: 2 })

Аttentive reader could notice, that such data loading is not optimal. Batch load by providing the array of userIds to the backend would be better. This example just demonstrates the async tasks. This is just a tool, how and when to use it depends on you.

⚙️ Additional array methods.

There are some other convenient methods to work with arrays. Check the Documentation

The methods I use often are:

• groupBy(user => user.score) Returns stream of groups ({ key, values: T[] }[]) grouped by the same value, by the score in this example.

• distinctBy(user => user.city) Returns stream of unique items by the field value. In this example: a user per city.

• sortBy(user => user.age, 'asc') Returns sorted stream. Possible directions asc and desc. For better text sorting there is sortByLocalCompare method.

• toMap(user => user.id, user => user.name) Returns an Map. From the example, the keys would be the ID of a user, and the value would be the name.

  I use this also to improve performance. Imagine, you need to query the name of a user by id often. Then instead of users.find(x =>x.id === id)?.name I create a Map of id:name and then simply get the name: map.get(id)

  toDictionary method returns object instead of Map, just in case it is more convinient for you

Happy coding

🏁

If you develop several projects you should consider to use private domains, e.g. https://foo.local , instead of e.g. http://localhost:12345.

Why?

• Obviously: you don't have to remember ports.
• Passwords: as you don't mess with ports, your dev passwords always stick to specific domains and is much better to manage them in password manager.
• Bookmarks: if you save links to fast-access them on your needs, you have much better meaningful links.
• Further more: by separation naming and deployment you are on a good way, therefor you may find other useful cases in your dev env.
• Certificates: you can setup this also with localhost, but anyways this step is a must, as many of browser features are HTTPS-only, like: WebRTC, Geolocation, Service Worker.

Is cross-platform: window, linux, mac. But was tested on windows. Required preinstalls: node, openssl, nginx.

On windows I would suggest to use https://community.chocolatey.org/packages to install all this dependencies.

1. Hosts

To make it simple, use npm:hostile

# install globally 
npm i hostile -g
# define your custom private domain
hostile set 127.0.0.1 foo.local

# view list
hostile list

2. Self-signed certificate

2.1 Create a certificate authority. This could be one time action - for any further domains it can be reused

# Generate private key
openssl genrsa -des3 -out myCA.key 2048
# Generate root certificate
openssl req -x509 -new -nodes -key myCA.key -sha256 -days 825 -out myCA.pem

2.2 Create a certificate for the domain.

Create a configuration file foo.local.ext

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = foo.local

Now you can create the cert using certificate authority keys:

# Generate a private key
openssl genrsa -out foo.local.key 2048
# Create a certificate-signing request
openssl req -new -key foo.local.key -out foo.local.csr
# Create the signed certificate
openssl x509 -req -in foo.local.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out foo.local.crt -days 825 -sha256 -extfile foo.local.ext

3. Install self-signed certificate

You can use Chrome to install those certificates, go to: chrome://settings/security

1. Import foo.local.crt to the Trusted Root Certification.
2. Import myCA.pem to the Trusted Publishers, if not already done

4. Nginx

Locate the configuration file, and add reverse proxy to your localhost development instance.

server {
    listen 443 ssl;
    server_name  foo.local;

    ssl_certificate      /path/to/foo.local.crt;
    ssl_certificate_key  /path/to/foo.local.key;

    location / {
        proxy_pass       http://localhost:5003;
        proxy_set_header Host $host;
    }
}
server {
    listen 80;
    server_name  foo.local;
    return 301 https://foo.local;
}

This also forces redirects to https.

Looking forward to improve:

It would be nice to have a tool which can automate all this steps and display better overview of already created projects. Also I would wish this tool to be able to launch my development environment for a project.

After trying to answer the question at stackexchange I've decided to write this short article, as the topic is indeed quite interesting.

1. Archive Node

The most simple is the last part of the question: at a specific block (onchain). As you may know, ethereum's geth node can be launched with various --gcmode values: full or archive. The archive mode doesn't remove any historical state data, it means - we can perform read queries specifying the blockNumber and getting the result back, as it was at that block.

Node-as-a-Service

All popular node providers, like infura.io, quicknode.com, alchemy.com can give you access to the archive node, but it won't be free of charge.

Self Hosting

As already mentioned, if you use geth include additionally --syncmode full --gcmode archive flags. But you have to be patient - it takes time to full sync the node, and it will occupy >10.2TB storage: etherscan/chainarchive

Erigon 🔗

I use erigon as it allows us to specify the number of blocks we want to keep with the historical state data

./erigon.exe --datadir D:/Erigon --chain mainnet --private.api.addr=127.0.0.1:9090 --prune=hrtc --prune.h.older=400000 --prune.r.older=400000 --prune.t.older=400000 --prune.c.older=400000

This will keep 400000 blocks, which with the average blocktime of 13s will keep the data for 2 months, and it takes 680GB of storage.

Intermediate summary

When you perform READ actions at blockchain you always get the result back which refers to some blockNumber. Even if you don't specify the block number, you get the result just for the latest block, but you can set any previous block your node supports.

2. Token Price

To retrieve the price onchain, there are 2 options - a) oracles: 3rd party contracts, which accumulate the prices in the correct way, and b) direct DEX sources - like Uniswap.

2.1 Oracles

The most convenient way, as it provides direct TOKEN/USD pair access, and most correct way, as it provides the most accurate price information at a given time point (blockNumber).

I suggest the ⬡ Chainlink price feed contracts: data.chain.link/crypto-usd. For an example, lets get the ETH price from the feed:

data.chain.link/eth-usd (0x5f4ec3df9cbd43714fe2740f5e3616155c5b8419)

Here I will use 0xweb 📦 package manager to create contract classes and to read data from the blockchain.

# install 0xweb as global util
$ npm i 0xweb -g
# initialize dependencies in the current folder, for API usage. For CLI not required 
$ 0xweb init

# download and generate classes for the contract
$ 0xweb install 0x5f4ec3df9cbd43714fe2740f5e3616155c5b8419 --name chainlink/oracle-eth

After the contract classes are generated, you can access onchain data via CLI or API

❗❣️❗ There are default KEYs for etherscan/co and infura. They are rate-limited. Please, create and insert your keys: 0xweb config -e, replace Node URLs for eth.

• cli

$ 0xweb contract read chainlink/oracle-eth latestAnswer

• api

import { ChainlinkOracleEth } from './0xweb/eth/chainlink/oracle-eth/oracle-eth';
import { Config } from '@dequanto/Config';
import { $bigint } from '@dequanto/utils/$bigint';

async function example () {
    await Config.fetch();

    const oracle = new ChainlinkOracleEth();
    const decimals = await oracle.decimals();
    const price: bigint = await oracle.latestAnswer();

    console.log(`ETH Price: ${ $bigint.toEther(price, decimals) } USD`);
}
example();

These examples return current ETH price. How to get the price at a specific block? Easy, by defining the block number.

$ 0xweb contract read chainlink/oracle-eth latestAnswer --block 14450000

 const price: bigint = await oracle.forBlock(14450000).latestAnswer();

⚠️ Ethereum 🗄️node URLs in default configuration are not archive nodes, so it won't work, you should replace them with archive node URLs.

Intermediate summary

Though ⬡ Chainlink is the most accurate and easy way to get the prices, but it doesn't contain all those thousands of tokens out there. That's why let's have a look at DEXes.

2.2 DEX

If a token is traded, that means we can get the price from there. Looks simple, but has its caveats. First of all, I'll show how to get the prices from UniswapV2 using 0xweb cli and dequanto libraries, and afterward, I briefly explain how it works under the hood.

• cli

0xweb has already the built-in token command, which can retrieve prices from UniswapV2

$ 0xweb token price LINK
# or with block number (archive node required)
$ 0xweb token price LINK --block 14450000
# or by address
$ 0xweb token price 0x514910771af9ca656af840dff83e8264ecf986ca
# example of the command return  
Symbol   LINK
Address  0x514910771af9ca656af840dff83e8264ecf986ca
Decimals 18
Price    15.715912

• api

import di from 'a-di';
import { Config } from '@dequanto/Config';
import { PlatformFactory } from '@dequanto/chains/PlatformFactory';
import { TokenPriceService } from '@dequanto/tokens/TokenPriceService';

async function example () {
    // we need this once: to find and read all configurations
    await Config.fetch();

    let token = 'LINK';
    // any unknown tokens are also supported
    let token = { address: "0x12345abcd", decimals: 18 };
    let chain = await di.resolve(PlatformFactory).get('eth')
    let service = new TokenPriceService(chain.client, chain.explorer);
    let { price } = await service.getPrice(token, {
        block: 14450000
    });
    console.log(price);
}
example();

A couple of words about how to run the scripts from api examples. I use atma global package to run the scripts. It supports file middlewares, in particular the atma-loader-ts, it will compile, cache and execute the TS files on the fly: atma run ./example.ts. All required packages and loader configuration will be created on 0xweb init. But you can use any other loader/builder you already use for TypeScript.

⚙️ How it works

2.2.1 UniswapV2 interfaces

interface IUniswapV2Factory {
    function getPair(address tokenA, address tokenB) external view returns (address pair);
}
interface IUniswapV2Pair {
    function token0() external view returns (address);
    function token1() external view returns (address);
    function getReserves() external view returns (uint112 reserve0, uint112 reserve1, uint32 blockTimestampLast);
}

1. How to get the price of a token e.g.WETH from the pair WETH/USDC?

This step is trivial, the pair, has reserves of each token - amount of tokens in the pool. And the price is just a ratio of these amounts. price(WETH) = totalEther(USDC)/totalEther(WETH).

• ⚠️ Be aware, that each ERC20 token can have different values for decimals, so just convert values to ether

• ✨ By having two tokens, e.g. 0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48(USDC)/0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2(WETH) which one is reserve0 and reserve1? Uniswap stores the tokens in a sorted way: token0 < token1

    const USDC = '0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48';
    const WETH = '0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2';
    const lpReserves = await poolPair
        .forBlock(opts?.block ?? opts?.date)
        .getReserves();
  
    let reserves = [lpReserves.reserve0, lpReserves.reserve1];
    let sorted = BigInt(USDC) < BigInt(WETH);
    if (sorted === false) {
        reserves.reverse();
    }
    let [ usdc, weth ] = reserves;
  

• ✨ From the previous example, you have noticed the .fromBlock(opts?.block ?? opts?.date) method - so yes, you can specify the BlockNumber or Date(which will be resolved to the BlockNumber).

2. Okay, but how do I find out the address of the pair contract?

The UniswapV2Factory contract has the method getPair, which accepts any two tokens, and returns the pair contract (if exists ❗).

// tokens order doesn't matter here
factoryContract.getPair(USDC, WETH)

3. What if I want to get the price for a token, but there is no TOKEN/USDC pair?

If we stick to on-chain solution only, you could index all pairs in the Factory, then you have the list of all pairs for a token. But this is unnecessary, as it is enough to check all major stable coins and the wrapped eth with that token: USDC, USDT, DAI, WETH.

• ⚠️ Often only TOKEN/WETH pair is present, or stable pairs have small liquidity, but as we already know - we can get the price for WETH from Chainlink or any of WETH/STABLE pairs

Following points you have to think about:

• ✨ to get the price from the pool with the most liquidity
• ✨ to get the price for one or two nearby blocks and take the average
• ✨ this gives the price in stable coins, those are not always equal to 1$

🏁