COMP5339 Data Engineering Notes by an HD Student | Weeks 1–11 | ETL, NoSQL, Kafka & Spark

Description

Written by a student who achieved a High Distinction (HD), this 33-page independent study guide turns the main COMP5339 Data Engineering Weeks 1–11 topics into clear, concept-focused revision notes. It covers: - data engineering foundations and lifecycle thinking - data acquisition, cleaning, ETL vs ELT, batch and streaming ingestion - databases, warehouses, lakes, lakehouses and OLAP - web scraping, REST APIs, XML, JSON and schema design - NoSQL systems, including MongoDB, Neo4j and wide-column NoSQL - spatial and temporal data engineering - stream processing, MQTT, Kafka, windows, watermarks and joins - scalability, row vs column stores, partitioning, sharding, replication and CAP - HDFS, MapReduce, Spark, Flink, DataOps, observability and ML systems The PDF includes diagrams, comparison tables, exam-style practice questions with model answers, marker notes, and an authored capstone practice scenario to help connect topics across the subject. Format: PDF, 33 pages. Scope: Weeks 1–11. Security/privacy material associated with Week 12 is intentionally excluded because it was not examinable in Semester 1, 2026. Independently authored study material. Not official University of Sydney material and not affiliated with or endorsed by the University. Check current subject and assessment requirements independently.