Streaming MySQL change sets to Kafka / AWS Kinesis

Streaming data changes happening in MySQL is a new subject but I’m thinking this particular way of capture change sets going to get used more widely with other data storages in future. AWS DynamoDB supports streams as of now without any middleware support.

I’ve used this CDC (Change Data Capture) streams in one of my previous projects in order to move events to Kafka for further processing, some people using similar behavior to invalidate caches as per changes happening in MySQL-table row level and etc.

I’ve used debebzium/Kafka Connect project for moving MySQL CDC events to Kafka which described in the following slide with more details.

For more information about Debezium visit http://debezium.io/.

Recently I’ve read an article about which you can achieve similar behavior in AWS RDS or standard MySQL instance and move CDC events to AWS Kinesis with minimal effort. It’s a great article, please take look!.

https://aws.amazon.com/blogs/database/streaming-changes-in-a-database-with-amazon-kinesis/

I think there are many possibilities opens up with particular technology/behavior, let me know if you think this could be useful for your future implementations or current business problem you’re trying resolve.