Release Notes: 7.0.0rc1

Welcome, developers! Today we introduce the next major release DipDup 7.0. After half a year of active development, the amount of new features is huge, but we hope that updated docs and lots of examples won't let you get lost.

Key highlights:

• Support for EVM-compatible blockchains
• Python 3.11 environment, updated Docker images
• Improved indexing performance
• Storage layer improvements
• Flexible deployment recipes for Docker and Compose
• New convenient CLI commands

Join our socials to discuss this release and ask any questions!

Twitter | Discord | GitHub

EVM support

Now DipDup supports EVM-compatible blockchains in addition to Tezos. The new index allows you to process contract events from Ethereum, Binance Smart Chain, Polygon, and other EVM-compatible networks. DipDup fetches historical data from Subsquid Archives, real-time data from RPC nodes, and ABIs from Etherscan. All you need is to define an index in your config and implement handlers for each event.

We have two demo projects for EVM: a very basic USDt price indexer and a more complex one for Uniswap V3. Run the dipdup new command, select "EVM" on the first step, and choose a template to use.

Developer experience

Our interactive installer was updated for better compatibility with pyenv and other Python tooling. The new command for creating new projects was also updated. It has a more convenient interface and includes EVM templates.

# Install DipDup
curl -Lsf https://dipdup.io/install.py | python3

# Create a new project interactively
dipdup new

Package and config discovery was improved, so managing a project is a bit easier now. You can omit default the dipdup.y[a]ml filename; packages in the current workdir are also correctly discovered.

dipdup -c . -c configs/dipdup.sqlite.yaml config export

Installer commands were moved to dipdup self command group. There you can switch the release channel or uninstall DipDup at any time:

dipdup self install --version 7.0.0rc1
dipdup self update
dipdup self install --force --ref next
dipdup self uninstall

We now use PDM as a default package manager. It's a swiss-knife to deal with Python's packaging pain with great PEP compatibility and a lot of features. Also, it can run scripts from pyproject.toml as npm does. Scaffolded scripts were updated from Poetry + GNU Make combo to PDM metadata.

Environment and performance

DipDup projects now run on Python 3.11. This release introduces very significant performance improvements, and that thing is crucial for indexers. Make sure to have fun with the latest syntax like walrus operator and structural pattern matching.

DipDup now better pre-fetch and cache data during indexing to increase processing speed. Our ORM was also updated to improve query performance and schema uniformity between database engines, see below.

Docker images are now based on Debian 12. They are simple, secure and easy to extend - just run pip as a default user. Alpine images are no longer published, but migration should be seamless.

Check the Docker page in the docs.

DipDup ORM

We continue to improve our storage layer based on Tortoise ORM, which we refer to as DipDup ORM. Currently, it's not a separate library, but a huge patchset. 7.0 release brings improved query performance, better schema uniformity between database engines, and new field types.

Since 7.0 release project models must be defined like this (note the changed imports):

from dipdup import fields
from dipdup import models


class User(models.Model):
    name = fields.TextField(pk=True)

Check the Models page in the docs.

Project package

The project package structure was significantly updated and is now easier to extend. All sections are created on init and pre-filled with stubs.

The DipDup package now includes three new sections:

• models section replaces models.py module as a source of DipDup ORM models. You can use any structure inside; models will be discovered automatically.
• configs directory contains env-specific configs to combine with the root config. Use them to define different settings for different environments and make your config more readable and declarative.
• deploy directory contains Dockerfile, Compose stacks and other deployment recipes. Also, there are .env.default files for each config in the configs directory. So you'll never miss a variable again!

The new dipdup package tree command allows inspecting the package structure and ensuring that everything is in place.

Check the Package page.

Miscellaneous

Here are some other notable changes not covered above:

• dipdup_meta internal table was added to the schema. You can store arbitrary JSON there and use it in your code. Survives reindexing. See the Internal tables page.
• Multiple feature flags were added for experimental and rarely used features. See the Feature flags page.
• Saved crash- and performance reports in the home directory can be viewed with new report and report show commands. Also, the crash_reporting flag (disabled by default) was removed from the config. If you want to report an issue and share the report, just drop us a message on GitHub or Discord.
• You can better customize logging levels in config. See Logging page.

Migration from 6.5

Perform the following steps to migrate your project to 7.0:

• Update your environment to Python 3.11. Install DipDup 7.0.0rc1 using our installer or manually.
• Run dipdup new command, choose empty template demo_blank.
• Move your old config to the new project. Update it using the guide below.
• Run dipdup init command. It will generate a new package structure. Commit the changes.
• Move your code from existing hooks and handlers to the freshly generated stubs. Keep an eye on imports and model attributes.
• Move SQL scripts, GraphQL queries and other metadata.
• Run pdm all to run linters and formatters. See pyproject.toml for scripts.

Config changes

• Set spec_version to 2.0.
• Add kind: tezos or kind: evm to contract definitions.
• Update kind field of index and datasource configs according to the table below.
• Remove advanced.crash_reporting and advanced.metadata_interface feature flags from your config if presented.