Introducing Sidetrek
Hi everyone!
I’m excited to introduce Sidetrek, an open-source CLI tool that lets you build an OSS modern data stack in minutes.
Key Features
- Fast project setup (~2-5 minutes): With just a couple of CLI commands, you can set up and run an end-to-end data project in minutes.
- Modern data tooling: Sidetrek’s current data stack is built on open-source modern data tools: Dagster (orchestrator), Meltano (ingestion), Iceberg (storage), DBT (transformation), Trino (query engine), and Superset (visualization).
- Smooth integration: All tools are specifically selected to play well together. They are pre-installed and pre-connected for you so you can get started right away.
- Isolated dev environment: The entire data pipeline runs in the local dev environment for fast iteration.
- Future expansion: We’ll be adding more tools for different use cases in the future.
Why?
It’s still too difficult to start data engineering projects. This is particularly true with the modern data stack, which is modular (awesome), but also makes the developer experience very fragmented (not so awesome).
Picking the right tools that work well together, setting up the tools, and then connecting them together to work seamlessly is challenging and time-consuming.
Who is it for?
As of today, Sidetrek is for greenfield data projects that cater to batch analytics use cases.
In the future, we’ll be adding data stacks for streaming, ML, and AI use cases as well as expanding the tooling options for each stack so you can work with your choice of tools.
We’ll also be modularizing the setup so it can fit into an already existing setup seamlessly.
Our Vision
Sidetrek’s ultimate goal is to bring an amazing developer experience for data engineers so they can iterate quickly locally, deploy with confidence, and focus on their core work rather than fiddling with tooling.
We think this is the future of the modern data stack: modular tools seamlessly connected together for a coherent developer experience.
This release is the first step towards this vision.
We’d love to get your feedback!
Please watch the quick demo video below if you’re interested.
We’d love for you to try it out and share your feedback!
Demo video: https://youtu.be/mSarAb60fMg
Website: https://sidetrek.com
Documentation: https://docs.sidetrek.com
Thank you!
Thank you for checking out Sidetrek, and we can’t wait to hear what you think!
Supported Platforms
Please note that it currently only works on Mac and Linux.
P.S.
If you’re not familiar with all the tools in our data stack, the demo can seem a little overwhelming. Don’t worry - we’ll be publishing an in-depth blog series over the next few weeks detailing each step of the data pipeline.
Please stay tuned!