Apache Azkaban with Postgresql

Running Apache Azkaban with Postgresql was a blocker for us that made us think about switching to Airflow(not planned so quick). I will cover the following topics explaining how we were able to run Azkaban with Postgresql -

1. Problem Statement
2. What did we do?
3. Resources for quick reference

Problem Statement

We were gladly using Azkaban with MySQL for quite some time and then came the requirement to have Postgresql and

Azkaban does not support Postgresql

We were left with 2 options, either bring our own Scheduler or introduce support with Airflow. Both were asking a good amount of time.

What did we do?

Then we searched Google and landed on this pull request (hot candidate to get merged in master) by mactaggart.
I forked the repo, get the code from the branch postgres-backend, build the docker image, did some configurations, and boom, it worked

Some points to know:

1. This version of Azkaban does not support the single executor mode
2. You need to activate the executor(at least I had to do it every time the Azkaban pod restarts)

Docker Image: Create your docker image from README.md of repo

Configuration (under conf directory)

A. In the executor and web configuration, add/update the below properties

database.type=postgresql
postgresql.port=<POSTGRESQL_PORT>
postgresql.host=<POSTGRESQL_HOST>
postgresql.database=<POSTGRESQL_DB>
postgresql.user=<POSTGRESQL_USER>
postgresql.password=<POSTGRESQL_PASSWORD>
postgresql.numconnections=100
postgresql.url.suffix=

B. In the web configuration set the below property to true

azkaban.use.multiple.executors=true

C. Execute all the Postgresql tables from https://github.com/mactaggart/azkaban/blob/postgres-backend/azkaban-db/src/main/sql/create.quartz-tables-all-postgresql.sql

Note: Please use the base image for your docker image considering the vulnerability

Resources for quick reference

Pull request — https://medium.com/r/?url=https%3A%2F%2Fgithub.com%2Fazkaban%2Fazkaban%2Fpull%2F2472

Code for build — https://medium.com/r/?url=https%3A%2F%2Fgithub.com%2Fmactaggart%2Fazkaban%2Ftree%2Fpostgres-backend

Docker Image — https://github.com/sunilkrg/AzkabanWithPostgresqlDocker

Please comment if you find any difficulty, I would be happy to help.