penguin-datalayer-core

Available Languages

The penguin-datalayer-core is the validation engine for the penguin-datalayer modules that belongs to raft-suite ecosystem created by DP6 in order to guarantee the Data Quality of the Data Engineering projects implemented on our clients, which is achieved through monitoring and automated data pipelines to guarantee credibility, consistency and availability in the planning, collection and data maintenance steps.

1. Requirements for usage

1.1 JSON Schema

The JSON Schema is a structure that allow the validation of JSON documents. This structure is used in the project because it allows the declaration of expected data formats in the data layer.

Supported Data Types

The following Data Types are currently supported within this module:

String
Number
Boolean
Object
Array

Validation rules

The following validation rules are accepted:

Enum (Equals): Should be used to validate the parity between the schema value versus the value sent to the data layer;
Pattern (Regex - String): Create regular expressions to validate the values of each key;
minItems (Array): Validates the minimum itens contained in an array;
Required: Should be used when a given key is required in the data layer

JSON Schema Structure

The following structure is a JSON Schema example:

{
  "$schema": "",
  "title": "Schema example",
  "array": {
    "$id": "#/properties/schema",
    "type": "array",
    "items": [
      {
        "type": "object",
        "properties": {
          "event": {
            "type": "string",
            "enum": ["test"]
          },
          "key1": {
            "type": "object",
            "properties": {
              "key1_sub1": {
                "type": "number",
                "enum": [10]
              },
              "key1_sub2": {
                "type": "string",
                "pattern": "teste|test|.*"
              },
              "key1_sub3": {
                "type": "string",
                "enum": ["production"]
              },
              "key1_sub4": {
                "type": "boolean",
                "enum": "desktop|mobile|msite"
              }
            },
            "required": ["key1_sub1", "key1_sub2", "key1_sub3", "key1_sub4"]
          }
        },
        "required": ["event"]
      }
    ]
  }
}

1.2 Ludwig (schema generator)

DP6 owns an automation that generates a structured validation schema.

Api Docs

How to contribute

Pull requests welcome! We would love some help to evolve this module. Feel free to search for open issues. If there’s a new feature or bug, please open a new issue, so our team can follow up.

Prerequirements

It will only be accepted contributions that follows the below requirements:

Padrão de commit

Support:

DP6 Koopa-troopa Team

e-mail: koopas@dp6.com.br