Objective

Design and implement Medallion Architecture on stock data, from source to data analytics.

Stock data introduction

Stock-related data* is generated every second from multiple sources, including users, the market, and stock exchanges. The Bronze Layer captures raw data, consisting of user transactions, market information, and stock and company registration details from the stock exchange.

After aggregation and cleaning, the Silver Layer contains structured and detailed data at a granular level, making it suitable for further processing.

In the Gold Layer, data is aggregated into daily, monthly, and yearly summaries, optimizing it for reporting, dashboards, and business intelligence applications.

*The data used in this project is artificially generated and does not represent real financial data. However, it is designed to closely mimic real-world scenarios, ensuring that stock prices fluctuate within a reasonable range over time.

Requirements and Grading Rubric (50 points)

1. Load data/stock_market.sql into MySQL database (5 points)

   1. Instructions can be found in project README

2. Load data/transactions.json into Mongodb (5 points)

   1. Instructions can be found in project README

3. Extract data from mysql and mongodb database(10 points)

   • Load data into spark sessions (5 points)

4. Aggregate data together for further data analysis (15 points) Note: only pyspark functions are allowed for data aggregation, other python packages such as pandas may not be used

   1. The final table should contain all information shown below, stock price fluctuates and the table requires average stock prices used for transactions, the volume of the transactions (sum of buy and sell), and the market index. Column sequence and name should follow the example given below.
   2. For further data analysis, aggregate the data into different granular level (see example below)

      1. Hourly data (4)

         

      2. Daily data (4)

         

      3. Monthly data (4)

         

      4. Quarterly Data (Oct - Dec Summary) (3)

         

5. Project Demo (15 points)

   • Be able to answer questions around project implementation
   • Create a merge request from your branch
   • Successfully pass all CI checks
   • Clear commit messages and merge request summary

Project will also be evaluated on (details will be released soon):

• Code quality (Function length & comment)
  • all functions have to have proper comments
  • All functions should be no longer than 50 rows (including comments)
• Commit message & merge request

Important Notes

• Demo Date: February 13th, in-class
  • No show on demo day will only receive a portion of the project 1 part 2 grade (50%)
• Submission deadline: February 14th, 1:00 AM (latest GitLab commit timestamp)