Table of Contents

Quickstart

This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details.

Let's create an indexer for the tzBTC FA1.2 token contract. Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.

demo_token demo is discussed in this guide; you can find it in the list of available templates when running dipdup new command.

A modern Linux/macOS distribution with Python 3.11 installed is required to run DipDup.

Create a new project

Interactively (recommended)

You can initialize a hello-world project interactively by choosing configuration options in the terminal. The following command will install DipDup for the current user:

Terminal

curl -Lsf https://dipdup.io/install.py | python3.11

Now, let's create a new project:

Terminal

dipdup new

Follow the instructions; the project will be created in the current directory. You can skip reading the rest of this page and slap dipdup run instead.

From scratch

If you don't want to use our installer, you can install DipDup manually. You can use any Python package manager you like, but we recommend PDM.

Terminal

# Create a new project directory
mkdir dipdup_indexer; cd dipdup_indexer

# PDM
pdm init --python ">=3.11,<3.12"
pdm add dipdup
pdm venv activate

# Plain pip
python -m venv .venv
. .venv/bin/activate
pip install dipdup

# or Poetry
poetry init --python ">=3.11,<3.12"
poetry add dipdup
poetry shell

Write a configuration file

DipDup configuration is stored in YAML files of a specific format. Create a new file named dipdup.yaml in the project root with the following content:

dipdup.yaml

spec_version: 2.0
package: demo_token

database:
  kind: sqlite
  path: demo_token.sqlite3

contracts:
  tzbtc_mainnet:
    kind: tezos
    address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
    typename: tzbtc

datasources:
  tzkt_mainnet:
    kind: tezos.tzkt
    url: https://api.tzkt.io

indexes:
  tzbtc_holders_mainnet:
    template: tzbtc_holders
    values:
      contract: tzbtc_mainnet
      datasource: tzkt_mainnet

templates:
  tzbtc_holders:
    kind: tezos.tzkt.operations
    datasource: <datasource>
    contracts:
      - <contract>
    handlers:
      - callback: on_transfer
        pattern:
          - destination: <contract>
            entrypoint: transfer
      - callback: on_mint
        pattern:
          - destination: <contract>
            entrypoint: mint

Generate types and stubs

Now it's time to generate typeclasses and callback stubs. Run the following command:

Terminal

dipdup init

DipDup will create a Python package demo_token having the following structure:

demo_token [src/demo_token]
├── abi
├── configs
│   ├── dipdup.compose.yaml
│   ├── dipdup.sqlite.yaml
│   ├── dipdup.swarm.yaml
│   └── replay.yaml
├── deploy
│   ├── .env.default
│   ├── Dockerfile
│   ├── compose.sqlite.yaml
│   ├── compose.swarm.yaml
│   ├── compose.yaml
│   ├── sqlite.env.default
│   └── swarm.env.default
├── graphql
├── handlers
│   ├── on_balance_update.py
│   ├── on_mint.py
│   └── on_transfer.py
├── hasura
├── hooks
│   ├── on_index_rollback.py
│   ├── on_reindex.py
│   ├── on_restart.py
│   └── on_synchronized.py
├── models
│   └── __init__.py
├── sql
├── types
│   ├── tzbtc/tezos_parameters/mint.py
│   ├── tzbtc/tezos_parameters/transfer.py
│   └── tzbtc/tezos_storage.py
├── py.typed
├── __init__.py
└── pyproject.toml

That's a lot of files and directories! But don't worry, we will need only models and handlers sections in this guide.

Define data models

Our schema will consist of a single model Holder having several fields:

address — account address
balance — in tzBTC
volume — total transfer/mint amount bypassed
tx_count — number of transfers/mints
last_seen — time of the last transfer/mint

Put the following content in the models/__init__.py file:

models/__init__.py

from dipdup import fields
from dipdup.models import Model


class Holder(Model):
    address = fields.TextField(pk=True)
    balance = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    turnover = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    tx_count = fields.BigIntField(default=0)
    last_seen = fields.DatetimeField(null=True)

Implement handlers

Everything's ready to implement an actual indexer logic.

Our task is to index all the balance updates, so we'll start with a helper method to handle them. Create a file named on_balance_update.py in the handlers package with the following content:

handlers/on_balance_update.py

from datetime import datetime
from decimal import Decimal

import demo_token.models as models


async def on_balance_update(
    address: str,
    balance_update: Decimal,
    timestamp: datetime,
) -> None:
    holder, _ = await models.Holder.get_or_create(address=address)
    holder.balance += balance_update
    holder.turnover += abs(balance_update)
    holder.tx_count += 1
    holder.last_seen = timestamp
    await holder.save()

Three methods of tzBTC contract can alter token balances — transfer, mint, and burn. The last one is omitted in this tutorial for simplicity. Edit corresponding handlers to call the on_balance_update method with data from matched operations:

handlers/on_transfer.py

from decimal import Decimal

from demo_token.handlers.on_balance_update import on_balance_update
from demo_token.types.tzbtc.tezos_parameters.transfer import TransferParameter
from demo_token.types.tzbtc.tezos_storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos_tzkt import TzktTransaction


async def on_transfer(
    ctx: HandlerContext,
    transfer: TzktTransaction[TransferParameter, TzbtcStorage],
) -> None:
    if transfer.parameter.from_ == transfer.parameter.to:
        # NOTE: Internal tzBTC transfer
        return

    amount = Decimal(transfer.parameter.value) / (10**8)
    await on_balance_update(
        address=transfer.parameter.from_,
        balance_update=-amount,
        timestamp=transfer.data.timestamp,
    )
    await on_balance_update(
        address=transfer.parameter.to,
        balance_update=amount,
        timestamp=transfer.data.timestamp,
    )

handlers/on_mint.py

from decimal import Decimal

from demo_token.handlers.on_balance_update import on_balance_update
from demo_token.types.tzbtc.tezos_parameters.mint import MintParameter
from demo_token.types.tzbtc.tezos_storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models.tezos_tzkt import TzktTransaction


async def on_mint(
    ctx: HandlerContext,
    mint: TzktTransaction[MintParameter, TzbtcStorage],
) -> None:
    amount = Decimal(mint.parameter.value) / (10**8)
    await on_balance_update(
        address=mint.parameter.to,
        balance_update=amount,
        timestamp=mint.data.timestamp,
    )

And that's all! We can run the indexer now.

Next steps

Run the indexer in-memory:

dipdup run

Store data in SQLite database:

dipdup -c . -c configs/dipdup.sqlite.yaml run

Or spawn a docker-compose stack:

cd deploy
cp .env.default .env
# Edit .env before running
docker-compose up

DipDup will fetch all the historical data and then switch to realtime updates. Your application data has been successfully indexed!

NextInstallation

Help and tips -> Join our Discord

Ideas or suggestions -> Issue Tracker

DipDup

—ChangelogContributionLicense