Architecture Guide - CaO_Data

Introduction

Kodda is a data integration platform built from several components working together around a centralized metadata repository: a web-based design and management interface, in-memory agents that do the work, a scheduler, a logging service and a message bus that connects them.

Architecture Overview

The architecture is organized around a modular metadata repository, which is accessed in application-server mode by the Kodda web interface and by the agents. All components can run on a single node, or the agents can be spread across many nodes to scale throughput.

Component	Role
Kodda Server	The web application that provides the design and management interface and the REST API.
Metadata Repository	A PostgreSQL database (`metaloader`) that stores connections, mappings, workflows, schedules, run history, published documentation and API keys.
Message Bus	A RabbitMQ broker. The server places work packages on queues; agents consume them. A second virtual host carries real-time status messages back to the browser.
Agents	Celery worker processes that execute the work: generating mappings, extracting data, loading targets, running workflows, browsing data.
Scheduler	A standalone service that fires scheduled jobs (One Time, Interval and Cron) by submitting mappings and workflows to the agents.
Log Server	A central logging service that captures the log files for every mapping and workflow run, so logs are available in one place however many agent nodes are in use.

Kodda Server

The Kodda server is an application server that provides the web-based design and management interface. It runs as the kodda_web service, with nginx in front of it handling browser connections. In a single-node configuration the same machine also hosts the metadata repository, message bus, agents, scheduler and log server.

Kodda Metadata

Kodda uses a centralized metadata repository to store the details of every object you create — connections, mappings, workflows, scheduled jobs — together with execution history and incremental load state. Every component connects to the repository using the [MetaDB] settings in setup.cfg.

Kodda Agents

Agents are lightweight in-memory worker processes, each dedicated to one kind of task. The Kodda server sends work packages (including the source and target connection details) to the agents over the message bus ([AgentQueue] in setup.cfg), and each agent picks up work from its own queue.

Kodda Scheduler

The scheduler is an optional standalone service (kodda_scheduler) that manages the timing and recurrence of mappings and workflows defined in the Scheduler screen. It listens on the host and port set in [SCHEDULER] in setup.cfg (default port 5500).

Kodda Logging

The log server (kodda_logging) captures the transactional history and log output for the whole Kodda environment. Because every agent writes to it, the Log links in Map History and Workflow History work regardless of which node ran the job.

Managing the Kodda Services

On Red Hat Enterprise Linux the installer registers each Kodda component as a systemd service, enables it to start at boot, and configures it to restart automatically if it stops unexpectedly. The services run as the kodda user.

Service	Component
`kodda_web`	Kodda Server (web interface and REST API)
`kodda_scheduler`	Scheduler
`kodda_logging`	Log Server
`kodda_d`	Default Agent
`kodda_genmap`	Generate Agent
`kodda_workflow`	Workflow Agent
`kodda_runmap`	Mapping Agent
`kodda_data_extract`	Data Extractor Agent
`nginx`	Web front end for `kodda_web`

To stop, start or restart all of them together, run the scripts in bin/rhel/ as root (or with sudo):

sudo bin/rhel/stop_kodda_services.sh
sudo bin/rhel/start_kodda_services.sh
sudo bin/rhel/restart_kodda_services.sh

To manage or check a single component, use systemctl with the service name:

sudo systemctl restart kodda_runmap.service
systemctl status kodda_data_extract.service

The message bus (rabbitmq-server) and the metadata database are separate services and are not stopped or started by these scripts.

Kodda Agents and Scaling

Overview

Kodda supports a multi-node architecture that greatly enhances the performance of both mappings and workflows. To scale, agents are installed and run on additional physical or virtual nodes, all connected to the same message bus and metadata repository.

Agent Scaling

By running standalone agents on separate nodes, Kodda takes advantage of the additional I/O, memory and CPU of those nodes to maximize throughput from source to target. Agents on different nodes share the same queues, so work is balanced across them automatically.

Agent Types

Each agent type listens on its own queue and runs as its own service:

Agent	Service	Queue	Default concurrency	Description
Default Agent	`kodda_d`	`default`	8	Browses source and target data (Browse Data, previews) and tests connections.
Generate Agent	`kodda_genmap`	`genmap`	4	Executes the Generate Mapping function.
Workflow Agent	`kodda_workflow`	`runwf`	8	Executes workflow management tasks: starting the mappings, scripts and emails in a workflow and tracking their completion.
Mapping Agent	`kodda_runmap`	`runmap`	8	Executes the target side of a mapping: DDL, staging and loading data into the target.
Data Extractor Agent	`kodda_data_extract`	`run_node`	8	Executes data extraction from source systems. A mapping with several Extractors uses several of these in parallel.

Agent Tuning

To maximize performance in a multi-node environment, focus on scaling the Data Extractor agents first, followed by Mapping agents and then Workflow agents as needed. Running Data Extractor and Mapping agents on as many nodes as possible is the first step in tuning Kodda. Agents are Celery workers, and each agent’s worker options are set in /etc/sysconfig/kodda:

Setting	Agent
`CELERYD_DEFAULT_OPTS`	Default Agent
`CELERYD_GM_OPTS`	Generate Agent
`CELERYD_WF_OPTS`	Workflow Agent
`CELERYD_RM_OPTS`	Mapping Agent
`CELERYD_DE_OPTS`	Data Extractor Agent

For example, the Data Extractor agent ships with:

CELERYD_DE_OPTS="--concurrency=8 -Q run_node -Ofair --max-tasks-per-child=1"

Two options to consider:

--concurrency=8 — the number of worker processes the agent runs. Each process handles one task at a time.
--autoscale=10,3 — use in place of --concurrency. The agent adds processes (up to 10) when there is work to do and removes them (down to 3) when the workload is low. Autoscaling can be slightly slower to respond than fixed concurrency, because starting a process takes time, but it saves resources while idle.

Leave the -Q queue name unchanged. After editing the file, restart the agent’s service for the change to take effect:

sudo systemctl restart kodda_data_extract.service

More agent processes are usually better, but there is a point beyond which adding processes hurts performance. Several smaller agents can also perform better than one large one — for example three agents with 10 processes each may outperform a single agent with 30 — because each agent has its own connection to the message bus and repository. Experiment to find the numbers that work best for your workload, task run times, hardware and source and target systems.

​Introduction

​Architecture Overview

​Kodda Server

​Kodda Metadata

​Kodda Agents

​Kodda Scheduler

​Kodda Logging

​Managing the Kodda Services

​Kodda Agents and Scaling

​Overview

​Agent Scaling

​Agent Types

​Agent Tuning