Data-warehousing

A data pipeline for Extracting, Loading and Transforming traffic data

Data Engineering TrafficDrone Data

A ELT pipeline using mysql, dbt, Apache Airflow, and Redash.

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. License
4. Acknowledgements

About The Project

A ELT pipeline with MySql for data storage, Airflow for automation and orchestration, DBT for data transformation, and a Redash dashboard connected to the data warehouse.

Built With

Tech Stack used in this project

• MySQL
• Apache Airflow
• dbt
• Redash

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• Docker Compose

Installation

1. Clone the repo

   git clone https://github.com/Samrawit02/Data-warehousing.git

2. Run

    docker-compose up

3. Open Airflow web browser

   Navigate to `http://localhost:8081/` on the browser
   

License

Distributed under the MIT License. See LICENSE for more information.

Acknowledgements

• 10 Academy