Subcribe and Access : 5200+ FREE Videos and 21+ Subjects Like CRT, SoftSkills, JAVA, Hadoop, Microsoft .NET, Testing Tools etc..

Batch Date: Aug 29th & 30th @8:30AM

Faculty: Mr. Vishwa (12+ Yrs Of Exp,..)

Duration: 8 Weekends Batch

Venue :
DURGA SOFTWARE SOLUTIONS,
Flat No : 202, 2nd Floor,
HUDA Maitrivanam,
Ameerpet, Hyderabad - 500038

Ph.No: +91 - 8885252627, 9246212143, 80 96 96 96 96

Syllabus:

Apache Kafka, ZooKeeper & Amazon MSK Masterclass - Real Time Streaming from Beginner to Advanced

Course Index

Weekend	Saturday	Sunday
1	Kafka Fundamentals	Kafka Architecture
2	ZooKeeper Fundamentals	ZooKeeper Administration
3	Kafka Installation	Kafka Configuration
4	Topics & Partitions	Replication & ISR
5	Producer Architecture	Producer Development Labs
6	Consumers	Consumer Groups & Rebalancing
7	Kafka Administration	Cluster Operations
8	Kafka Security	Monitoring & Troubleshooting
9	Kafka Connect	Kafka Streams
10	AWS MSK	MSK Security & Operations
11	Confluent Platform	Production Architecture
12	Terraform	GitHub Actions + Capstone

WEEKEND 1 — KAFKA FUNDAMENTALS

Saturday — Introduction

• What is Apache Kafka?
• Why Kafka?
• Messaging vs event streaming
• Traditional messaging architecture
• Kafka use cases
• Kafka vs RabbitMQ, ActiveMQ, SQS, Kinesis and Pulsar
• Synchronous vs asynchronous communication
• Event-driven architecture
• Pub/Sub architecture

Sunday — Kafka Architecture

• Producer, Consumer, Broker, Topic, Partition, Replica
• Leader, Follower, ISR
• Consumer Group, Offset, Controller, Cluster
• Create first topic and produce/consume messages
• Lab: first single-node Kafka deployment and message flow

WEEKEND 2 — ZOOKEEPER

Saturday — ZooKeeper Fundamentals

• What is ZooKeeper?
• Why Kafka historically used ZooKeeper?
• Ensemble, Leader, Follower, Quorum
• ZNode, ephemeral and persistent nodes
• Watches, sessions and leader election
  

Sunday — ZooKeeper Administration

• zkServer start/stop/status
• ZooKeeper shell and ZNode navigation
• tickTime, initLimit, syncLimit
• clientPort and dataDir
• Logs, backup, recovery and troubleshooting
• Lab: 3-node ZooKeeper ensemble

WEEKEND 3 — KAFKA INSTALLATION & CONFIGURATION

Saturday — Installation

• Linux/Amazon Linux/Ubuntu installation
• Java and environment variables
• Kafka directory structure
• Kafka services and startup/shutdown
• Lab: Kafka on EC2
  

Sunday — Configuration

• broker.id
• listeners
• advertised.listeners
• log.dirs
• num.partitions
• default.replication.factor
• min.insync.replicas
• zookeeper.connect
• Lab: build a 3-broker cluster
  

WEEKEND 4 — TOPICS, PARTITIONS & REPLICATION

Saturday — Topics & Partitions

• Topic lifecycle
• Partitioning and partition keys
• Ordering guarantees
• Partition assignment
• Kafka topic CLI: create, list, describe, alter and delete
  

Sunday — Replication

• Replication factor
• Leader and followers
• ISR
• Out-of-sync replicas
• Leader election
• Preferred replica election
• Unclean leader election
• Lab: kill a broker and observe failover

WEEKEND 5 — PRODUCERS

Saturday — Producer Architecture

• ProducerRecord
• Serializer
• Partitioner
• Batching
• Buffering
• Compression
• acks, retries, batch.size, linger.ms, buffer.memory, compression.type
  

Sunday — Producer Labs

• Console producer
• Java producer
• String and JSON serialization
• Key-based partitioning
• Custom partitioner
• Lab: produce events to multiple partitions

WEEKEND 6 — CONSUMERS & CONSUMER GROUPS

Saturday — Consumers

• Consumer architecture
• Consumer Group
• Group Coordinator
• Partition assignment
• Offsets
• Auto commit vs manual commit
• Console and Java consumer labs
  

Sunday — Consumer Groups

• Consumer scaling
• Rebalancing
• Partition ownership
• Static membership
• Cooperative rebalancing
• Lab: scale consumers, stop consumers and observe rebalances

WEEKEND 7 — KAFKA ADMINISTRATION

Saturday — Administration

• Broker administration
• Topic administration
• Partition management
• Replica management
• Log retention
• Log segments
• log.retention.hours
• log.retention.bytes
• log.segment.bytes
  

Sunday — Cluster Operations

• Add/remove brokers
• Partition reassignment
• Replica movement
• Cluster expansion/shrink
• Rolling restart
• Broker maintenance
• Lab: add Broker 4 and rebalance

WEEKEND 8 - SECURITY, MONITORING & TROUBLESHOOTING

Saturday — Security

• TLS/SSL
• SASL
• SASL/PLAIN
• SASL/SCRAM
• Kerberos concepts
• Kafka ACLs
• Principals, resources and operations
• kafka-acls.sh lab
  

Sunday — Monitoring

• JMX
• Prometheus
• Grafana
• Consumer lag
• Under-replicated partitions
• Offline partitions
• Request latency
• Throughput
• Disk utilization
• Important metrics: UnderReplicatedPartitions, OfflinePartitionsCount, ConsumerLag, BytesIn, BytesOut

WEEKEND 9 — KAFKA CONNECT & KAFKA STREAMS

Saturday — Kafka Connect

• Connect architecture
• Source vs Sink connectors
• Worker, Connector and Task
• Standalone vs Distributed mode
• File source lab
• JDBC source/sink lab
  

Sunday — Kafka Streams

• KStreams and KTables
• Stateless and stateful processing
• Windowing
• Aggregation
• Joins
• Lab: orders stream  aggregation  output topic

WEEKEND 10 — AWS MSK

Saturday — Amazon MSK

• What is Amazon MSK?
• MSK vs self-managed Kafka
• MSK Provisioned vs Serverless
• Broker architecture
• VPC, subnets and security groups
• Multi-AZ architecture
  

Sunday — MSK Security & Operations

• IAM authentication
• TLS
• SASL/SCRAM
• Encryption at rest/in transit
• KMS
• CloudWatch
• MSK monitoring
• Scaling and configuration
• Lab: deploy and connect to MSK

WEEKEND 11 — CONFLUENT & PRODUCTION ARCHITECTURE

Saturday — Confluent

• Confluent Platform
• Confluent Cloud
• Control Center
• Schema Registry
• Kafka Connect
• ksqlDB
• REST Proxy
• Event streaming architecture
  

Sunday — Production Architecture

• Multi-AZ Kafka
• Multi-region Kafka
• Disaster recovery
• Backup and restore
• MirrorMaker 2
• Cluster Linking
• RPO/RTO
• Capacity planning
• Storage/network sizing

WEEKEND 12 — TERRAFORM, CI/CD & CAPSTONE

Saturday — Terraform

• Terraform provider and resources
• VPC
• Subnets
• Security Groups
• EC2
• User data
• ZooKeeper deployment
• Kafka broker deployment
• terraform init/fmt/validate/plan/apply/destroy
• Remote state with S3 and state locking
  

Sunday — GitHub Actions + Capstone

• CI: fmt, validate, plan
• Checkov/tfsec security scanning
• AWS OIDC
• IAM role
• Environments: dev/stage/prod
• Approval gates
• Terraform apply
• Deployment testing
• Final project: AWS EC2 Kafka + ZooKeeper + Terraform + GitHub Actions + monitoring

FINAL CAPSTONE PROJECT

• Build a 3-broker Kafka cluster on AWS EC2.
• Deploy a ZooKeeper ensemble for the ZooKeeper-based Kafka lab.
• Create topics with multiple partitions and replication factor 3.
• Build producer and consumer applications.
• Test consumer groups, offsets, lag and rebalancing.
• Perform broker failure and verify leader failover.
• Configure Kafka security and ACLs.
• Add Prometheus/Grafana monitoring.
• Provision infrastructure entirely with Terraform.
• Use GitHub Actions for validation, security scanning, plan and apply.
• Use GitHub OIDC with an AWS IAM role rather than long-lived AWS keys.
• Document architecture, operations, troubleshooting and disaster recovery.
  

Target Architecture

Layer	Components
Source Control / CI/CD	GitHub + GitHub Actions + OIDC
Infrastructure	Terraform + AWS VPC + Security Groups
Compute	EC2 Kafka brokers + ZooKeeper ensemble
Streaming	Apache Kafka topics, partitions, replicas
Applications	Producer + Consumer + Consumer Groups
Integration	Kafka Connect + Schema Registry
Processing	Kafka Streams / ksqlDB
Cloud Managed	AWS MSK
Operations	Prometheus + Grafana + CloudWatch
Advanced	MirrorMaker 2 / Cluster Linking / DR

Learning Outcomes

• Kafka Developer: build producers, consumers, consumer groups, serializers and Kafka Streams applications.
• Kafka Administrator: install, configure, secure, monitor, scale and troubleshoot Kafka clusters.
• Cloud Engineer: deploy Kafka on EC2 and AWS MSK with AWS networking and security.
• DevOps Engineer: automate infrastructure with Terraform and CI/CD with GitHub Actions.
• Architect: design highly available, multi-AZ, multi-region and disaster-recovery Kafka platforms.