Skip to end of metadata
Go to start of metadata

PNDA-4032 - Getting issue details... STATUS

Motivation


Following problems exist at present -

  • /etc/hosts inflexible to changes and integration with external discovery
  • Endpoints change over time but nothing updates them
  • Hand rolled implementation for endpoint registrar (on DM)
  • Configuration inflexibly handled as multiple config files on various nodes

Proposal

Establish a clearer pattern for the following, based on names and dynamic registration rather than addresses & static configuration.

  • Internal node name registration
  • Internal service name registration (for internal wiring)
  • External service name registration (for PNDA service discovery outside the perimeter)

Definitions

Node

  • Names given to nodes rather than services for private access within PNDA
  • Format: <hostname/minion_id>.node.<data _centre>.<domain>
  • e.g. pnda123-hadoop-edge.node.dc1.pnda.local
  • PNDA controls the naming

Internal service

  • Names given to services rather than nodes for private access within PNDA
  • Format: <service_name>service.<data_centre>.<domain>
  • e.g. opentsdb-internal.service.dc1.pnda.local
  • PNDA controls the naming

External services 

  • Names given to services rather than nodes for public access outside PNDA
  • Format: anything
  • e.g. opentsdb-ingest.pnda.acme.org
  • PNDA operator controls the naming

Deployment

These things happen automatically.

Node

Internal services

External services

Configuration

There's nothing to do for nodes and internal services. PNDA handles this automatically.

Setting FQDNs for external services

In normal usage, external names and the secure certificates that go with them will be supplied by the PNDA operator and then used on the PNDA perimeter.

Ground truth for external FQDN to external service mapping is provided to PNDA using this mechanism -

Implementation notes

  • If you need to use the true external FQDN for a service, look it up from the pillar
  • Important: never assume anything about DNS authority for FQDNs, sometimes it will be in Consul if nothing else is provided, but it might not be.
  • Code inserts into Pillar the names created by the pnda-cli for Consul, if nothing else is provided
  • UIs in PNDA are injected with FQDNs so as to redirect correctly

Discovery

  • Console to update to use service registry to decide what to show
  • Now possible/sensible to render 'live' PNDA topology, for example
  • DM to update to use service registry and to inject appropriate name aliases into applications e.g. kafka.pnda.local for discovery, broker1.kafka.pnda.local for specific broker, etc
  • All components to update to use names/aliases from service registry

Configuration

  • Configuration files to migrate to dedicated config service
  • All components to update to use config service

Health

  • Platform testing to make use of service availability in health assessments

Provisioning

  • Remove grain based address resolution & hard wiring of endpoints
  • Remove hard wiring of config files through pillar for some data, make dynamic

Other

  • Config and service registry need to be available, persistent, recoverable

Plan

  • Agree on paper what names/aliases/endpoints are needed and scheme for organizing in DNS: see Names / aliases / endpoints for more information (done)
  • Diagram what solution will look like (done)
  • Add name service and populate with current contents of /etc/hosts (done)
    • At this point PNDA can be integrated with other systems with 'static' endpoints
  • Remove /etc/hosts code and replace with configuration of name service client (done)
  • Service [de]registration and DM injection of these endpoints into apps (25% done)
    • Kafka
    • Relevant Hadoop endpoints e.g. name service
    • OpenTSDB
  • Use of registered endpoints in PNDA services and components (need a list to work through) (25% done)
    • Example: OpenTSDB will use HBase endpoint by name rather than address by role grain
  • Agree on paper what config should be dynamic and scheme for storing in hierarchy. see Service configuration and discovery for more information 
  • Populate KV store for config (need a list to work through)
  • Use of config from KV store in PNDA services and componentds (need a list to work through)
  • Remove redundant grain based wiring of config and endpoints
  • Use endpoint availabilty in health assessments
  • Use topology information directly in console

Interfaces

  • DM interface to retrieve endpoints to be deprecated
  • New interfaces to be documented for retrieving endpoints and config
  • DNS for PNDA cluster available to clients/external DNS

Compatability

  • DM interface to retrieve endpoints to be deprecated
  • Applications using properties correctly simply need [re]redeployment
  • No labels