DuneSQL is Dune's future, and will soon be the sole query engine. Here's what this means for you, and how we'll tackle the migration together.
A few weeks ago, DuneSQL became the default query engine on Dune.com.
We’ve seen impressive early adoption: 85% of all newly created queries are on DuneSQL and 30% of all executions are on DuneSQL. You can read all about DuneSQL in the blogpost from our CTO.
DuneSQL is Dune’s future. Therefore, we will be progressively sunsetting both SparkSQL and Ethereum V1 (Postgres) to make DuneSQL the only query engine available on Dune.
Both SparkSQL and our old Postgres systems are not optimal for querying crypto data. Sunsetting these systems is an inconvenient but necessary step to make Dune the best platform to query crypto data.
We understand that this transition might be challenging, especially if you've previously migrated from V1 to SparkSQL. We empathize with the inconvenience this migration may cause and want to assure you that this is an essential step towards simplifying the user experience on Dune and focusing our engineering resources on a single, powerful query engine.
To smoothen the transition to DuneSQL we are investing heavily in migration support to make the process as seamless as possible.
We are focusing our migration efforts on:
The high-level timeline for SparkSQL and PostgreSQL along with FAQ are available now in our docs here: https://dune.com/docs/query/Old-Query-Engines
Data ingestion for Polygon, BNB, Optimism, and Gnosis on their respective Postgres instances have stopped on 20/04/2023.
This means that all DuneV1 Postgres queries on these systems will no longer be updated. All queries on V2 (Spark, DuneSQL) querying this data will continue to function normally.
We recently released the migrate to DuneSQL tool that automatically converts queries from SparkSQL and PostgreSQL to DuneSQL. We have open sourced it to allow the community to provide feedback and contribute. We are also working on a GPT4 integration to make the tool even more powerful!
Please refer to the docs to keep up to date with the progress: https://dune.com/docs/query/Old-Query-Engines/migration-tool/
We are aware that DuneSQL is still running into some trouble with advanced queries that query many different tables, most often resulting in stage-related timeouts.
We are working on addressing this issue on our side by materializing more tables and optimizing our data model. Furthermore, we are actively working on a feature that will allow you to easily use query results as materialized views and the introduction of query scheduling.
These new features should help avoid stage timeouts by allowing you to build mini-data pipelines within Dune.
For the short-term future, Spells will keep running on SparkSQL, but queryable by DuneSQL. We are already testing a new version of Spellbook that will run entirely on DuneSQL, closing the last remaining compatibility gaps and building a simpler Spell development journey.
Once Spellbook on DuneSQL has been fully validated, all Spells will be migrated by the Dune Team.
We appreciate your continued trust in us, and we hope you are as excited as us about the possibilities of DuneSQL.
Please accept our sincere apologies for the inconvenience caused by the migration. Rest assured this is the final migration, DuneSQL is our definitive answer as to how we can build the best crypto data platform!
Thank you for your understanding and support. Don’t hesitate to reach out to us via email@example.com for any questions or concerns.
The data must flow!
We hit day 30 of our 90 day DuneSQL Migration Plan, here's how it's going so far
At Dune we’re building the world’s first community data platform. A big part of that is Spellbook, our community data pipeline.
The revolution will not be reported quarterly.
Celebrating Dune wizards. Dashboards, stories, alpha dropped in your inbox weekly.