#Pentaho Data Integration Variables Full List Here

Basically we keep track of all fields as they are created, split, aggregated, transformed, etc. Data lineage is an oft-overloaded term, but for the purposes of this blog I will be talking about the flow of data from external sources into steps/entries and possibly out to other external targets. With visual tools to eliminate coding and complexity, It puts the best quality data at the fingertips of IT and the business.In Pentaho Data Integration 6.0, we released a great new capability to collect data lineage for PDI transformations and jobs. System RequirementsWhat is Pentaho Data Integration and what are its top alternatives It enable users to ingest, blend, cleanse and prepare diverse data from any source. Write Variables to Log Step Write Variables to Log Job Entry Write Result Rows to Log Job Entry Write Result Files to Log Job Entry Send Log to Variable Job Entry Developed by Inquidia Consulting. The Log Helper plugin for Pentaho Data Integration provides several steps and job entries to assist with common logging tasks.

Variables (class, age and gender) and, for each item, calculate the.The code for the current data lineage capability is entirely open source and is available on GitHub here. Alert(getVariable('NUMBEROFDAYS', '')) Both transformations use the same steps, but they have separate task.More precisely, we present the Pentaho Data Integration (PDI-CE)2, called also. So in that sense you can follow your data all the way through your PDI process.1) Set variable transformation is used to set the value: var numberOfDays 100 Alert(numberOfDays) setVariable('NUMBEROFDAYS', numberOfDays, 'r') 2) Get variable transformatoin only reads it. When jobs or transformations call other jobs or transformations, that relationship is also captured.

This lets the step report its own outgoing fields, and many times the step needs to know the incoming fields before it can properly report the output fields. For example, getting the list of fields that are output from a transformation step involves calling the getStepFields() API method. This includes APIs that are more general than uniform, as flexibility has seemed a more important goal than introspection. However PDI is a very flexible and powerful tool.

It is possible to parameterize things like table names, field names, etc. This requires a valid database connection and introduces extra latency into the lineage collection.Other considerations include variables and parameters. In the case of a Table Input step, the query is actually executed so the metadata of the ResultSet is available to determine the output fields.

We use Apache Tinkerpop 2.6 for this (more on that in a moment)- If a step/entry is data-driven, we collect lineage information as each row is processed. There is one caveat to this that I'll talk about at the end of the blog, but it is unsupported, undocumented, and more geared for the community for the time being.At present, the architecture and flow for collecting lineage is as follows:- Just before a transformation is executed, a graph model is created and associated with the running Trans object. For this reason, data lineage collection is performed at run-time, such that the fields and values are all resolved and known. Even using parameters' default values doesn't work as the default value may be meant to fail the transformation (to ensure valid parameter values are passed in).

Pentaho Data Integration Variables Full List Here

This is the entry point to be called before a transformation starts, namely at the TransformationStartThreads extension point (see full list here). These (when present for a step/entry analyzer) are called by the lineage collector to get relationships between steps (and/or their fields) to resources outside the transformation, such as text files, databases, etc.The workhorse in this situation (the 'lineage collector') is implemented in TransformationRuntimeExtensionPoint (there's a Job version too of course). To report full lineage, a step/entry needs a specific analyzer that understands the business logic of that step or entry.- There are 'decorator' analyzer interfaces (and base implementations) that can be associated with step/entry analyzers, most notably ExternalResourceStepAnalyzer and StepDatabaseConnectionAnalyzer (and the JobEntry versions thereof). - hasn't changed as a result of the logic) and any new fields have no relationships to the incoming fields (as we don't know what those relationships are). So all fields are assumed to be pass-through (meaning the structure of a named field - type, e.g.

This may be tied to the lineage graph someday (and indeed there is some data common to both) but for now it is there to collect something like the PDI Operations Mart and logging do, but in a uniform fashion with a standard format (JSON).PDI's data lineage model is based on a property graph model. The top-level analyzer adds its own lineage/metadata to the graph, then iterates over the steps/entries' so their analyzers can add their lineage information (see Graph Model and Usage below)NOTE: In the code you will see lots of references to ExecutionProfile. The Runnable calls into the parent entity's analyzer, so a TransformationAnalyzer or JobAnalyzer. The method also creates a future runner for the lineage analysis, which will be called when the transformation is complete.The transFinished() method spins off a new thread to perform the full lineage analysis. When a graph model is created, nodes for all the concepts (transformation, job, database connection, step, field, etc.) are added to the graph as well. This provides a well-known entry point for querying the graph if no other information is known.

Also the model includes 'concept nodes' that allow for more targeted graph queries. For our model, the nodes are things like the executed jobs/transformations, steps, stream fields, database connections, etc. Nodes and edges can have properties (such as name = 'my transformation' or relationship = 'knows').

However 2.x had more than enough functionality for us, we just had to bring in the pieces we needed. Having said that, Tinkerpop 3 requires Java 8, and since PDI 6.0 supports Java 7, we had to use the deprecated 2.6 version. The 3.x line of Tinkerpop has been accepted to the Apache Incubator, and I certainly congratulate them on that achievement! Tinkerpop 3.x has absorbed all the 2.x products into a single product, and represents an impressive leap forward in terms of graph planning/processing engines. This query returns nodes corresponding to all transformations executed for this lineage artifact.For our property graph model implementation, we chose the open-source project Tinkerpop. In our example, we could use it to start a query from the Transformation concept node and find all nodes connected to it via an outgoing 'parentconcept' edge. In practice, it is a 'parentconcept' edge from the concept node to the instance(s) of that concept.

There is some overhead involved with this ORM (due to the amount of reflection and such that is needed), so we only use Frames at present for integration testing. It allows the lineage code to offer a Java method whose body is essentially a Gremlin query that returns the specified object. Graph queries in Pentaho data lineage are materialized as Gremlin statements, which are executed as a process graph using the Pipes frameworkFrames: This is basically an ORM from the graph/process models to Java objects. The process pipeline(s) are themselves modelled as a graph (called a process graph)Gremlin: This is the actual data traversal language, available as a Java API as well as a fluent Groovy DSL.