Webinar
The Real-time Data Engineering Series
How to Build and Fix CDC Pipelines the Right Way
Dani Pálma, Staff Engineer
Rob Meyer, VP Marketing
Change data capture (CDC) is usually considered to be one of the hardest parts to implement in a data pipeline. Many data engineers have trouble trusting CDC for this reason. But CDC is battle-tested and proven. If you implement CDC the right way, it’s not only easy, it’s also incredibly reliable.
Join resident Join us as we through lessons learned the hard way from real-world projects on how to implement CDC the best way, and what to do when CDC goes wrong.
By the end of this session you’ll have learned:
- How CDC works
- How to build a CDC pipeline the right way, and the wrong way
- The top challenges with CDC, and how to fix them, including:
- Different CDC and snapshotting options compared
- Source and destination failures, and how to avoid data loss
- Fan-out streaming architectures
- Replication with or without primary keys
- Schema evolution techniques
We'll will go through tips and tricks for different databases and different failures with demos, and how to best build your pipelines to reduce or avoid these challenges.