Skip to end of metadata
Go to start of metadata

Entity relationships

PNDA application management provides interfaces for users to create an application and manage its lifecycle. A PNDA application is consists of one or more component instances, which is of different types and can be long-lived or short-lived. In order to abstract common states and transitions at application level, we need understand what key manageable entities are and what their relations are.

Packages are independently deployable units of application layer functionality. Each package consists of definitions of one or more components, each of which has a defined type (e.g. upstart, Oozie workflow, Oozie coordinator, or Jupyter notebook).

A component instance can be long-lived (e.g. Oozie coordinator job) to schedule and control the execution of another component instance (e.g. Oozie workflow job) when a per-defined condition is met (e.g. when certain dataset becomes available) or periodically before ending time. A component instance can be a short lived process that spawns a single or a workflow of jobs. In this case, the component instances is a scheduler. A component instance itself can be an atomic job, what can either short-lived or long-lived too. Depending on underlying technologies, a job can be a Hadoop MapReduce job, or YARN application, or Spark streaming job.

An application therefore can be conceptualized as an instance of a package and consists of one or more component instances (i.e. instantialization of components).


Component FSM

Based on reviewing states and transitions of different component types, a FSM is defined at component abstract level. 

Submitting an one-shot or long-running component instance

The component instance will be created but not started. It will be in PREP state.

Starting an one-shot or long-running component instance

The component instance will be created and it will start. The component instance will be in RUNNING state. 

Pause a long-running component instance

The component instance (e.g. Oozie coordinator job) will be in PAUSED state. However scheduled subflow jobs (e.g. Oozie actions) will remain in current state (e.g. RUNNING until they finish successfully) before transiting to PAUSED state.

Resume a long-running component instance

The component instance (e.g. Oozie coordinator job) will be in RUNNING state. However jobs (e.g. workflow jobs it scheduled) will remain in current state (e.g. PAUSED until they are scheduled to run) before transiting to RUNNING state.

STOP an one-shot or long-running component instance

When an component instance finishes execution successfully, it will be in STOPPED state. 

Kill an one-shot or long-running component instance

The component instance will be in KILLED state. All spowned subflow jobs will be killed and moved to KILLED state. 


  • No labels