ETL using Python: from MySQL to BigQuery

This is a direct and to the point course that will get you quickly ETL’ing data from MySQL to BigQuery.

The lessons in this course are broken out into short How-Tos. Therefore you can take this course over the weekend and be ready to show off your skills on Monday morning!

Things that we will cover:

• Setup
  • Setting up a GCP Account
  • Credential and Authentication for security
  • Python Environment Setup
• Extract
  • Use Python to connect to MySQL
  • Use Python’s pandas to export data
  • Python library usage for saving files to file paths
• Transform
  • Use Python functions to transform data
  • Learn how to use Python pandas to transform data
  • Use inline SQL during Extract for data transformation
• Load
  • Use the BigQuery Python library
  • Connect to BigQuery
  • Load data to BigQuery
  • Incremental Loads vs Truncate and Load
  • Other data handling options during Load

After taking this course, you’ll be comfortable with the following pretty cool things:

• Connect to MySQL using Python
• Learn how to obscure your database credentials so you’re not exposing them in your code
• Usage of the os module for the purpose of saving files and hard coding fewer things.
• Use both Python and the pandas library to transform data on the fly during the Transformation phase of your ETL
• Learn how to use GBQ’s modules/libraries to make the loading of the data a very easy, straightforward task

Have fun, enjoy and keep growing!