Officially out now: The TypeDB 3.0 Roadmap >>

Data pipelines

This reference covers the usage of data pipelines in TypeQL. Data pipeline read and modify data in stages.

Pipeline kinds

  • A write pipeline may comprise stages that modify data in the database (such as insert or delete).

  • A read pipeline only comprises stages that read data from the database but do not modify it.

Data manipulation stages

Match stage

Used to match existing data instances in the database.

Fetch stage

Used to format the output of a data pipeline into JSON.

Insert stage

Used to add new data instances into the database.

Delete stage

Used to remove existing data instances from the database.

Update stage

Used to modify existing data instances in the database.

Put stage

Used to add new data instances to the database if they do not already exist.

Stream manipulation stages

Select operator

Used to keep specified variables for each element of the data stream and remove the rest.

Require stage

Used to remove elements from the data stream that do not contain specified optional variables.

Sort operator

Used to order the elements of the data stream based on the value of specified variables.

Limit operator

Used to keep a specified number of elements of the data stream and remove the rest.

Offset operator

Used to remove a specified number of elements from the data stream.

Reduce operator

Used to perform reduction operations on the data stream according to specified groupings.

Preamble stage

With stage

Used to define functions on an ad-hoc basis for use in the data pipeline.