Data Engineering Sensor Data

A fully dockerized ELT pipeline project, using MYSQL, dbt, Apache Airflow, and Redash.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgements

About The Project

Using a docker-compose file, developed a completely dockerized ELT pipeline with MySQL for data storage, Airflow for automation and orchestration, DBT for data transformation, and a Redash dashboard connected to the MySQL database.

Built With

Tech Stack used in this project

• MYSQL
• Apache Airflow
• dbt
• Redash

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• DockerCompose

Installation

1. Clone the repo

   git clone https://github.com/bereketkibru/Data_engineering_sensor_data

2. Run

    docker-compose build
    docker-compose up

3. Open Airflow web browser

   Navigate to `http://localhost:8000/` on the browser
   activate and trigger dbt_load_dag
   activate and trigger dbt_dbt_dag

4. Access redash dashboard

   Navigate to `http://localhost:5000/` on the browser

5. Access your mysql database using adminar

   Navigate to `http://localhost:8080/` on the browser
   choose mysql databse
   use `root` for username
   use `pssd` for password

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Bereket Kibru - @email - bekakibru2@gmail.com

Acknowledgements

• 10 Academy