Embedded Google PubSub Emulator for Testing

Recently I’ve been working with Google Cloud Platform PubSub for various Java applications and Apache Beam pipelines. PubSub offers effortless and cloud-native messaging topic for your software application or BigData processing requirements.

One of the problems I faced was to run PubSub emulator locally part of my Integration test suite. After hunting for ways to run PubSub emulator locally part integration test I’m happy to present my following code snippet whoever out there who looking for the same solution.

Dependencies

<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsub</artifactId>
<version>1.35.0</version>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>1.7.0</version>
</dependency>

Usage
EmbededPubSub embededPubSub = new EmbededPubSub();
embededPubSub.start();
embededPubSub.subscriber(message -> System.out.println(" @@ Received : " + message), 10000);
embededPubSub.publish("Hello Pub/Sub!");
embededPubSub.stop();

I’ve been utilising testcontainers to spin up a container which runs gcloud cli components and then exposes few ports for host computer to communicate through.

Next thing is if you ever needs to pull/browse messages/subscriptions or other information from Local Pub/Sub emulator for debugging purposes unfortunately we’re unable to use `gcloud` cli tools for this at the time of writing this blog post. workarounds are either write a client code or use Pub/Sub Rest API.

Pub/Sub Rest API

https://cloud.google.com/pubsub/docs/reference/rest/

I hope this article will be useful for somebody who couldn’t find their head around with Pub/Sub Integration test.

VirtualBox VM hard disk resize won’t work with Snapshots

Virtual Box Logo
I had a Windows7 guest machine running on Ubuntu 16.04, recently I’ve ran out space from my primary partition so I executed following command and Tried to extend partition size with Windows Disk Management area but it seems resize didn’t took a effect couldn’t find any unallocated amount with my primary partition.

Command to Resize .VDI
vboxmanage modifyhd "/home/kasun/VirtualBox VMs/windows/windows-disk1.vdi" --resize 200000
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Then i look around and found that VirtualBox VMs with snapshots needs resizing snapshot disk (current snapshot) in order take actual effect of it. Alternatively you could clone current state of machine to a new vm then execute above command to resize cloned disk.

Then using gparted or Windows Disk Management (Windows Guest only) tool you could extend your partition size.

Run a MySQL instance with docker

I wanted to use docker to run MySQL in isolation so i could change it’s version/state at anytime.  I see a special benefit when running MySQL or any application in a isolation where it does not depend around host machines file-system or Operating system, which means it allows us to run any available version instantly. Also they’re highly portable.

In this example I’ve used official docker-MySQL to run instances. Official image is flexible to configure around our specific needs and it’s been well documented.

Please refer to official docker-MySQL page for more configuration options.

 

Following Makefile allows to run persistent and non-persistent MySQL instances which opens up port 3306 on host machines to containerised MySQL instance. Current MySQL version  is 5.5 and default persistent directory been set. Feel free to change as required.

https://gist.github.com/KasunDon-awin/85c331de7f2f81579135530d51342764

Download above file and called it as mysql-server or any name you wanted. Then simply execute following commands in order to launch/stop/restart instances.

If you need persistent MySQL instance to be launched use

make -f mysql-server start-persistent

or use following for non-persistent instances

make -f mysql-server start

Stop or restarting instances can be done as below,
make -f mysql-server stop
make -f mysql-server restart

That’s all you can access your MySQL instance with your default mysql-client.

mysql -h 127.0.0.1 -u root

Sync to containers using File Watchers

I think most of us, using docker to run applications simultaneously in one host machine. May be some of you already using docker containers as development aid tool rather than using it’s production capabilities, it’s a bit not ideal when you have to build docker images each time you making a changes to your application sources while it’s running on docker,  It take some effort to rebuild and re-run containers,  Whereas we build images which are ready to deploy or make releases.

This is why i was looking for some syncing tool that sync changes to running docker containers simultaneously. There are lot of other method we could do similar task, but found way to use PHPStrom to do this task using it’s file watchers.

FileWatchers are pretty good feature comes with PHPStrom IDE which can be used for other similar where necessary,

Here’s how i did it..

First of all open up a project, then File -> Settings [Ctrl + Alt + S]

File_Settings

Then it’ll open up Settings window, choose Tools -> File Watchers

Tools_File_Watchers

Next Click on Green add (+) button on right top to add a new file watcher. There are predefined file watcher available to choose from but this instance we’ll choose custom so we can define our own file watcher rules.

choose_custom

 

Now New Watcher Dialog will appear on your screen, this is where we define routines for our own file watcher.

new_watcher

We can fill in watcher name and description fields to identify particular file watcher.

You can define any specific File Type to be watched and Program to be called if any changes been detected to any watching file type. Also you can add extra configurations options depending upon your needs. There are lot of predefined macro’s available to try out!.

Finally!, Select File Watcher and Press Apply and OK.

select_watcher

You’ll notice whenever you change watching file type it’ll automatically sync that file into docker container instantly. If there’s a error it’ll load up failed watcher command in console. so you can identify the problem.