Building AWS Glue ETL Pipelines with Python Shell for S3 and Redshift Integration

In this video, we will develop AWS Glue ETL script using Python shell. We can now use Python scripts in AWS Glue to run small to medium-sized ETL (extract, transform, and load) workflow. Previously, AWS Glue jobs were limited to Apache Spark environment.
Python shell jobs in AWS Glue support scripts that are compatible with Python 2 and 3 and come pre-loaded with libraries such as the Boto3, Numpy, SciPy, pandas, and others. We can also, install other libraries via .whl file.

Link to medium article on the topic: https://blog.devgenius.io/develop-aws-glue-etl-pipeline-with-python-shell-fe6f66763e9d

Link to code, GitHub: https://github.com/hnawaz007/pythondataanalysis/tree/main/AWS%20Glue%20ETL%20Python%20Shell

Subscribe to our channel:
https://www.youtube.com/c/HaqNawaz

---------------------------------------------
Follow me on social media!

GitHub: https://github.com/hnawaz007
Instagram: https://www.instagram.com/bi_insights_inc
LinkedIn: https://www.linkedin.com/in/haq-nawaz/

---------------------------------------------

#Python #ETL #AWS

Topics covered in this video:
0:00 - Introduction ETL with Python shell
0:53 - Pre-Requisites
1:30 - Create Python .whl file
2:35 - Python ETL script
4:15 - Upload scripts to AWS
5:11 - AWS Glue ETL Job
6:33 - AWS Redshift table
6:49 - Execute Glue ETL Job
7:17 - Review Data & logs