Building AWS Glue ETL Pipelines with Python Shell for S3 and Redshift Integration

In this video, we will develop AWS Glue ETL script using Python shell. We can now use Python scripts in AWS Glue to run small to medium-sized ETL (extract, transform, and load) workflow. Previously, AWS Glue jobs were limited to Apache Spark environment. Python shell jobs in AWS Glue support scripts that are compatible with Python 2 and 3 and come pre-loaded with libraries such as the Boto3, Numpy, SciPy, pandas, and others. We can also, install other libraries via .whl file. Link to medium article on the topic: https://blog.devgenius.io/develop-aws-glue-etl-pipeline-with-python-shell-fe6f66763e9d Link to code, GitHub: https://github.com/hnawaz007/pythondataanalysis/tree/main/AWS%20Glue%20ETL%20Python%20Shell Subscribe to our channel: https://www.youtube.com/c/HaqNawaz --------------------------------------------- Follow me on social media! GitHub: https://github.com/hnawaz007 Instagram: https://www.instagram.com/bi_insights_inc LinkedIn: https://www.linkedin.com/in/haq-nawaz/ --------------------------------------------- #Python #ETL #AWS Topics covered in this video: 0:00 - Introduction ETL with Python shell 0:53 - Pre-Requisites 1:30 - Create Python .whl file 2:35 - Python ETL script 4:15 - Upload scripts to AWS 5:11 - AWS Glue ETL Job 6:33 - AWS Redshift table 6:49 - Execute Glue ETL Job 7:17 - Review Data & logs