Embedded Google PubSub Emulator for Testing

Recently I’ve been working with Google Cloud Platform PubSub for various Java applications and Apache Beam pipelines. PubSub offers effortless and cloud-native messaging topic for your software application or BigData processing requirements.

One of the problems I faced was to run PubSub emulator locally part of my Integration test suite. After hunting for ways to run PubSub emulator locally part integration test I’m happy to present my following code snippet whoever out there who looking for the same solution.

Dependencies

<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsub</artifactId>
<version>1.35.0</version>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>1.7.0</version>
</dependency>

Usage
EmbededPubSub embededPubSub = new EmbededPubSub();
embededPubSub.start();
embededPubSub.subscriber(message -> System.out.println(" @@ Received : " + message), 10000);
embededPubSub.publish("Hello Pub/Sub!");
embededPubSub.stop();

I’ve been utilising testcontainers to spin up a container which runs gcloud cli components and then exposes few ports for host computer to communicate through.

Next thing is if you ever needs to pull/browse messages/subscriptions or other information from Local Pub/Sub emulator for debugging purposes unfortunately we’re unable to use `gcloud` cli tools for this at the time of writing this blog post. workarounds are either write a client code or use Pub/Sub Rest API.

Pub/Sub Rest API

https://cloud.google.com/pubsub/docs/reference/rest/

I hope this article will be useful for somebody who couldn’t find their head around with Pub/Sub Integration test.

Streaming MySQL change sets to Kafka / AWS Kinesis

Streaming data changes happening in MySQL is a new subject but I’m thinking this particular way of capture change sets going to get used more widely with other data storages in future. AWS DynamoDB supports streams as of now without any middleware support.

I’ve used this CDC (Change Data Capture) streams in one of my previous projects in order to move events to Kafka for further processing, some people using similar behavior to invalidate caches as per changes happening in MySQL-table row level and etc.

I’ve used debebzium/Kafka Connect project for moving MySQL CDC events to Kafka which described in the following slide with more details.

For more information about Debezium visit http://debezium.io/.

Recently I’ve read an article about which you can achieve similar behavior in AWS RDS or standard MySQL instance and move CDC events to AWS Kinesis with minimal effort. It’s a great article, please take look!.

https://aws.amazon.com/blogs/database/streaming-changes-in-a-database-with-amazon-kinesis/

I think there are many possibilities opens up with particular technology/behavior, let me know if you think this could be useful for your future implementations or current business problem you’re trying resolve.

Basic Docker swarm Operations

Docker swarm mode natively built on top of docker engine, So it’s easier to orchestrate and clustering capabilities on containers using docker swarm mode.

Following commands summarize basics of swarm mode operations.

Initialize a swarm
docker swarm init --advertise-addr <manager-ip-addr>

Swarm Status
docker info

List all swarm nodes
docker node ls

Get Swarm token to join as worker
docker swarm join-token worker

Get Swarm token to join as manager
docker swarm join-token manager

View all swarm services
docker service ls

Create a swarm service
docker service create --replicas 1 --name <name> <image> <command>

Inspect a running swarm service
docker service inspect --pretty <service-name>

Scale up/down a swarm service (increase/decrease replicas)
docker service scale <service-name>=<no-of-replicas>

Remove a swarm service
docker service rm <service-name>

Setup rolling update interval when creating service
docker service create --replicas 3 --name --update-delay 10s

Rolling Update running swarm service
docker service update --image <service-name>

Force Update stuck service updates without rolling manner
docker service update <service-name>

Filter swarm services by running state
docker service ps --filter="running" <service-name>

 

Hope this helps to get started with docker swarm. For more information:: https://docs.docker.com/engine/swarm/#feature-highlights

Enable MySQL Query Logs

I was using one of ORM tools inside my project suddenly I’ve faced a strange issue where it doesn’t do any interactions with MySQL. There’s NO errors on client side it seems queries never get executed so we wanted to listen connection/all queries coming to mysqld. I wanted to get a broader idea of what’s been going on.

I’ve checked MySQL manual and found concept of MySQL General Query Logging functionality which is spot-on for my requirement. I gave it a shot!.

This might be useful Anyone of you facing same situation as me.

MySQL General Query Log : https://dev.mysql.com/doc/refman/5.6/en/query-log.html

You can do this two ways,

1. Define in MySQL configuration file (my.cnf)
[mysqld]
general_log = on
general_log_file=/var/log/mysql/queries.log

You might need `mysqld` reloaded once add above lines of config.

2. MySQL Run-time via executing following queries
> set global general_log_file = "/var/log/mysql/queries.log";
> set global general_log = "ON";

Having Turn on MySQL query log might consume your hard drive space a lot. I would recommend to use whenever you needs it, Turn Off when you don’t needs it to avoid issues.

Personally I always recommend to use in Run-time so you can turn it off by executing following command.

> set global general_log = "OFF";

This might help!! Use it with caution.

VirtualBox VM hard disk resize won’t work with Snapshots

Virtual Box Logo
I had a Windows7 guest machine running on Ubuntu 16.04, recently I’ve ran out space from my primary partition so I executed following command and Tried to extend partition size with Windows Disk Management area but it seems resize didn’t took a effect couldn’t find any unallocated amount with my primary partition.

Command to Resize .VDI
vboxmanage modifyhd "/home/kasun/VirtualBox VMs/windows/windows-disk1.vdi" --resize 200000
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Then i look around and found that VirtualBox VMs with snapshots needs resizing snapshot disk (current snapshot) in order take actual effect of it. Alternatively you could clone current state of machine to a new vm then execute above command to resize cloned disk.

Then using gparted or Windows Disk Management (Windows Guest only) tool you could extend your partition size.

Apache sets default Response charset

I’ve been currently working on a application that should render charset ISO-8859-1 to the browser.  So I’ve added a following meta tag to HTML document header.

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

For some reason when rendering the page,  it always makes some characters broken or not been recognised. I’ve spend a good few hours to debug this situation thought it was due to some issue with application itself how it handles encoding.

Thanks to built-in browser “Inspect Element” tool, I’ve managed to crack down the root issue. It was causing this issue due to Response ContentType set to use charset UTF-8.

Content-Type:text/html; charset=utf-8

Application wasn’t setting ContentType charset in Response, Then where did this charset came from ?. HTML header meta tag wasn’t taking a effect in this case. It has overridden by Response ContentType charset.

-This is the time Apache took my attention.

It turned out that Apache was automatically sets AddDefaultCharset config directive to be UTF-8. It was automatically added to /etc/httpd/conf/httpd.conf by Default Installation.

AddDefaultCharset UTF-8

I’ve commented out  AddDefaultCharset directive form Apache config and reloaded Apache, it worked as expected.

I guess if you wanted to control encoding by application level then you should AddDefaultCharset Off or remove it from Apache config.

Hope this helps to solve unexpected encoding issues due to Apache config.

Backup MySQL to Google Drive

I always had this issue when i started to working on new projects where do i keep my MySQL backup data, if i keep backup data same server as my application running it’ll soon consume all disk space just for backup data. So I wanted to find a solution for this.

 

Cloud Hosting providers tend to charge extra fees if you want to enable automated backups or there might be terms which you don’t like. I think most of startups/individual projects don’t want to spend too much money on their backups or may be funding might be limited.

I’ve look around for cheap but reliable service for this, The solution I’ve came across is to use Google Drive as my backup storage which has fair prices to increase storage(if needed, otherwise stick to free storage limit), reliable and highly available. Also there’s a well documented API available for this service.

I’ve used python for all background work for this task, then it’ll upload file straight to Google Drive.

Installation and help guide available on https://github.com/KasunDon/google-drive-mysql-backup

Feel free to give me a feedback or report any issues.

Run a MySQL instance with docker

I wanted to use docker to run MySQL in isolation so i could change it’s version/state at anytime.  I see a special benefit when running MySQL or any application in a isolation where it does not depend around host machines file-system or Operating system, which means it allows us to run any available version instantly. Also they’re highly portable.

In this example I’ve used official docker-MySQL to run instances. Official image is flexible to configure around our specific needs and it’s been well documented.

Please refer to official docker-MySQL page for more configuration options.

 

Following Makefile allows to run persistent and non-persistent MySQL instances which opens up port 3306 on host machines to containerised MySQL instance. Current MySQL version  is 5.5 and default persistent directory been set. Feel free to change as required.

https://gist.github.com/KasunDon-awin/85c331de7f2f81579135530d51342764

Download above file and called it as mysql-server or any name you wanted. Then simply execute following commands in order to launch/stop/restart instances.

If you need persistent MySQL instance to be launched use

make -f mysql-server start-persistent

or use following for non-persistent instances

make -f mysql-server start

Stop or restarting instances can be done as below,
make -f mysql-server stop
make -f mysql-server restart

That’s all you can access your MySQL instance with your default mysql-client.

mysql -h 127.0.0.1 -u root

Login to AWS EC2 instances

Recently I’ve started to use AWS (Amazon Web Services) to host few of my web apps. I’ve had a little hard time to figure out how to access instances using *.pem file associated while i was creating AWS Instance.

First of all, locate pem file and assign appropriate permissions to it.


chmod 400 /path/my-key-pair.pem

Then depending upon OS (Operating System) of your ec2 instance you’ve created, you’ve to access appropriate user in order get into your ec2 instance.

Amazon Linuxec2-user
RHEL5root | ec2-user
Ubuntu – ubuntu
Fedora – fedora | ec2-user
SUSE Linux – root | ec2-user

All Others OS’s – root | ec2-user  or check with your AMI provider.

user_name@public_dns_name

For Example
ssh -i /path/my-key-pair.pem ec2-user@ec2-121.compute-1.amazonaws.com

More information please refer to AWS Official User Guide.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html

Sync to containers using File Watchers

I think most of us, using docker to run applications simultaneously in one host machine. May be some of you already using docker containers as development aid tool rather than using it’s production capabilities, it’s a bit not ideal when you have to build docker images each time you making a changes to your application sources while it’s running on docker,  It take some effort to rebuild and re-run containers,  Whereas we build images which are ready to deploy or make releases.

This is why i was looking for some syncing tool that sync changes to running docker containers simultaneously. There are lot of other method we could do similar task, but found way to use PHPStrom to do this task using it’s file watchers.

FileWatchers are pretty good feature comes with PHPStrom IDE which can be used for other similar where necessary,

Here’s how i did it..

First of all open up a project, then File -> Settings [Ctrl + Alt + S]

File_Settings

Then it’ll open up Settings window, choose Tools -> File Watchers

Tools_File_Watchers

Next Click on Green add (+) button on right top to add a new file watcher. There are predefined file watcher available to choose from but this instance we’ll choose custom so we can define our own file watcher rules.

choose_custom

 

Now New Watcher Dialog will appear on your screen, this is where we define routines for our own file watcher.

new_watcher

We can fill in watcher name and description fields to identify particular file watcher.

You can define any specific File Type to be watched and Program to be called if any changes been detected to any watching file type. Also you can add extra configurations options depending upon your needs. There are lot of predefined macro’s available to try out!.

Finally!, Select File Watcher and Press Apply and OK.

select_watcher

You’ll notice whenever you change watching file type it’ll automatically sync that file into docker container instantly. If there’s a error it’ll load up failed watcher command in console. so you can identify the problem.