Using Pipelines

A guide to pipelines in Simplebim Cloud: what a pipeline is, how it is created and given a dataflow, a trigger and file locations, how it is validated and run, what happens during a run, and how its status controls when it runs by itself.

Pipelines live on the Pipelines tab of a workspace, described in Managing Workspaces, which also covers the History tab where the runs of all pipelines are listed and handled. The connections that pipelines read from and write to are described in Configuring Connections. Building the dataflow itself is done in Simplebim desktop — see Using the Dataflow Palette — and it can be published straight into a pipeline from there; see Using the Simplebim Cloud Dialogs.

Contents

• What a pipeline is
• The Pipelines tab at a glance
• Setting up a pipeline
• Creating and renaming a pipeline
• The dataflow
  • Uploading a dataflow file
  • Using a dataflow from the library
  • The dataflow details
• The trigger
  • Defining the trigger
  • How the trigger works
  • Moving the trigger file after the run
• File locations
  • Mapping the file locations
  • The Select Folder dialog
• Validating a pipeline
• Running a pipeline manually
  • The Run Pipeline dialog
  • Following the run
• What happens during a run
• Pipeline status
• Cancelling running instances
• Messages
• Troubleshooting

What a pipeline is

A pipeline runs a dataflow in the cloud, automatically, every time a file arrives. It consists of four things:

Each run processes one trigger file: it is downloaded together with the other input files, the Simplebim Engine runs the dataflow, and the outputs are uploaded to the output locations. The run is recorded in the workspace's History with its log.

The Pipelines tab at a glance

Add New Pipeline at the top right creates a pipeline. Below it, one row per pipeline:

The three warnings — dataflow, trigger, file locations — are your checklist: when all three columns show green, the pipeline can run.

Creating and configuring pipelines requires account administrator rights. Members with access to the workspace see the pipelines, can open the dataflow details, and can Validate and Run.

Setting up a pipeline

The order that works:

1. Create the pipeline and give it a name. It starts Paused; nothing runs by itself until you activate it.
2. Add the dataflow — upload the .sbdf file from Simplebim desktop, publish it from the desktop's Cloud menu, or select it from the library. This tells the pipeline which input and output locations it needs.
3. Define the trigger — the folder to watch and the file filter.
4. Map the file locations — a connection and a folder for each input and output location.
5. Validate — every folder must be found.
6. Run manually with a test file and read the log in the History tab.
7. Activate the pipeline when it should react to files by itself.

Before step 3 the account needs at least one authorized connection, and it helps to have the workspace connections defined; see Using the Workspace Page.

Creating and renaming a pipeline

Add New Pipeline asks only for the Pipeline Name and creates the pipeline with Create. Names must be unique within the workspace, regardless of case; a duplicate is refused with A pipeline with this name already exists.

The pencil next to the name opens Edit Pipeline, where the Pipeline Name and a Description can be changed. The description is shown as the tooltip of the name. Save is enabled once something differs from the saved values.

The dataflow

The Dataflow column's + (no dataflow yet) or ⓘ (dataflow present) button opens the dataflow dialog — Add Dataflow - name or Dataflow - name. It shows the current dataflow's details, lets administrators upload a new file, and in library accounts offers the library actions.

Uploading a dataflow file

Under Upload a dataflow file drop a .sbdf file onto the drop zone, or click it to browse. The file is not stored yet: Simplebim Cloud examines it first and shows the result in the dialog:

• a green box The dataflow is valid for Cloud. with the dataflow's Name and Cost — the dataflow can be saved;
• a red box The dataflow is not valid for Cloud and cannot be saved: followed by the reasons — typically a step that reads a file on a local disk instead of a file location, or a tool the cloud does not have. Fix the dataflow in Simplebim desktop and upload again;
• possibly a yellow Warnings box with things worth knowing that do not prevent saving.

Write a Comment if you like — what changed, for whom — and click Save. The dataflow is stored on the pipeline and the message The dataflow was saved to the pipeline. appears. Cancel discards the staged file. Saving replaces the pipeline's earlier dataflow, and when the pipeline was following a library dataflow it is detached from the library; the dialog says both before you save.

Saving also updates the pipeline's file locations: an input or output location the new dataflow names for the first time is added unmapped, one it no longer names is removed, and one it still names keeps its connection and folder. Check the File Locations column after every upload.

The comment can also be edited on its own; the footer turns into Save / Cancel as soon as it differs from the stored one.

Using a dataflow from the library

In accounts that have the Dataflow Library — see Using the Dataflow Library for what it is and who may use what — the dialog also offers Select from library…. The Select Dataflow dialog asks for the Area — the account library or a workspace library you may read — the Dataflow, and the Version:

• Latest (updates when a new version is published) — the pipeline follows the dataflow: when a new version is published, the pipeline takes it into use at its next run;
• Specific version — the pipeline is pinned to the chosen version until you change it.

OK points the pipeline at the library dataflow, copies that version into the pipeline and syncs the file locations exactly as an upload does. The message Dataflow selected. confirms it.

When a newer version exists than the one a pinned pipeline uses, the grid shows the blue ↑ vN badge and the dialog offers Update to the latest version. For a pipeline that follows latest the same button re-fetches the newest version at once instead of waiting for the next run.

The dataflow details

The dialog shows, read-only:

The trigger

Defining the trigger

The pencil in the Trigger column opens Define Trigger - name:

Tick Move trigger file after successful pipeline run to reveal a second row with its own Connection and Move to Folder — the handled folder; see Moving the trigger file after the run. Untick it to stop moving files.

Save stores the trigger; the message Trigger was successfully saved appears and the Trigger column shows the green tick once connection, folder and filter are all set. A pipeline has one trigger folder; a dataflow that should react to files in several folders needs several pipelines.

How the trigger works

While the pipeline is Active, Simplebim Cloud polls the trigger folder every few minutes and looks for files that are new or changed since the previous poll and match the filter. Each such file starts one run with that file as the trigger file.

What happens to files that are already waiting in the folder when the trigger is armed — when the pipeline is activated, when its trigger or file locations are saved while it is active, and when the account's pipelines are flushed — depends on the handled-folder setting:

• a pipeline that moves trigger files after the run processes every waiting file that matches the filter, one run each;
• a pipeline without a handled folder starts watching from that moment and leaves the waiting files alone. Run those manually, or drop them into the folder again.

During a run a small lock file named after the trigger file — ending in .cloudjob — sits next to it in the trigger folder and disappears when the run ends. After a failed run a marker ending in .jobfailed is left next to the file so that a flush does not rerun a broken file endlessly. Deleting the marker, or uploading the file again, lets it run again.

A Paused pipeline does not react to files. When it is activated again the same rule applies: with a handled folder the files that arrived meanwhile are processed, without one only files that arrive from then on.

Moving the trigger file after the run

With Move trigger file after successful pipeline run ticked, a run that ends with Success or Had Warnings moves its trigger file from the trigger folder to the handled folder. The trigger folder then holds only files still waiting or failed, which makes it easy to see the state of a project, and a flush never reruns a handled file.

If the file cannot be moved — the handled folder was deleted, or is not writable — the run still counts as successful but is recorded as Had Warnings, with the reason in its log. A failed run never moves its file.

The handled folder has a second effect: a pipeline that moves its trigger files also processes the files already waiting in the trigger folder whenever its trigger is armed — on activation, after saving its trigger or file locations while active, and on Flush Pipelines. This is what makes a trigger folder behave like an in-tray: whatever is in it gets processed and moved out. Without a handled folder the pipeline only ever reacts to files that arrive after the trigger was armed.

File locations

A dataflow reads its inputs from and writes its outputs to named locations — for example Models and Export. Which names exist is decided in the dataflow; the pipeline's job is to say where each of them is in your storage. The File Locations column counts these locations and shows the green tick when every one of them has a connection and a folder.

Input locations also carry the file names the dataflow reads from them, so the run downloads only those. Output locations take whatever the dataflow writes.

Mapping the file locations

The pencil in the File Locations column opens Map File Locations - name, one row per location:

Save is enabled once something changed and stores all rows at once — File locations were successfully saved. The trigger and handled locations are not in this dialog; they belong to Define Trigger.

The trigger folder can double as an input location: point the input location whose file is the trigger file at the same connection and folder. Several locations may share a folder, and an output folder may be a subfolder of an input folder.

The Select Folder dialog

Select Folder in the trigger and file location dialogs opens a tree of the storage. Its root is the chosen connection's own folder — for a workspace connection, that connection's folder inside the account connection — and you cannot browse above it. Expand folders to load their subfolders, select one and click Select Folder. When the field already holds a folder the dialog opens with that folder selected; if it no longer exists, the nearest existing parent is selected and the dialog warns Current folder does not exist: followed by the old path.

Validating a pipeline

Validate opens Pipeline Validation - name and checks every location of the pipeline — trigger, inputs, outputs and the handled folder — one after the other:

The same checks run automatically at the start of every run; a run whose checks fail ends as Validation Failed without processing anything. Validating by hand before activating a pipeline saves you that failed run.

Running a pipeline manually

A manual run is the way to test a pipeline with a chosen file, and to process a file that was already in the folder. It is allowed while the pipeline is Active or Paused; an Archived pipeline cannot be run.

The Run Pipeline dialog

Run in the pipeline's row opens Run Pipeline with the Workspace and the Pipeline already selected; both can be changed to any non-archived workspace and pipeline you can see. The dialog then reads the trigger folder and offers under Select Trigger File (*.ifc) the files that match the trigger filter. If the folder cannot be read the reason is shown in a yellow box; if it is empty, There are no files in the trigger file location.; if nothing matches, There are no files matching the filter "…" in the trigger file location. The Open in … ↗ button opens the trigger folder so you can put a file there.

Pick the file and click Run. The dialog closes and either the message Pipeline run was started successfully. appears or, when the progress dialog is enabled for your installation, the Pipeline Run Progress dialog opens. If the pipeline cannot be started the reason is shown — for example that its status does not allow it.

Following the run

The Pipeline Run Progress dialog follows the run live: a status line such as Waiting in queue… or Processing…, the row of phases Validating → Waiting → Preparing → Processing → Finalizing → Finished with the time each took, the Files table as soon as files are transferred, and the Log growing at the bottom, with Scroll to Bottom when you have scrolled up. It ends with Completed successfully in …, Completed with warnings in … or Failed after …. Copy Log copies the log; Close closes the dialog without affecting the run. After thirty minutes the dialog stops following and tells you to check the History tab later.

Without the progress dialog, follow the run on the workspace's History tab: use the refresh button of the workspace row, and open the log with 📄 once the run is finished.

What happens during a run

Whether started by a file or by hand, a run goes through the same phases, all visible in the log:

Each run consumes the account's tokens according to the dataflow's cost. Members who should be notified when runs finish are set up on the Account page; see Using the Account Page.

Pipeline status

Delete replaces Archive in the menu for a pipeline that has never run; it asks for confirmation and cannot be undone. Once there is history, the pipeline can only be archived, which keeps the history readable.

Pausing takes effect immediately: the trigger is removed. Activating arms it again — see How the trigger works for what happens to files already waiting. Editing the trigger or the file locations of an Active pipeline re-arms the live trigger as you save, with the same effect.

Cancelling running instances

Account administrators see a red Cancel button in a pipeline's row while it has runs in progress — Cancel 3 when there are several. Clicking it asks for confirmation and cancels every running instance of that pipeline; the message 1 instance was cancelled. or N instances were cancelled. confirms it. Cancelled runs appear in the History as Cancelled; their trigger files stay in the trigger folder without a failure marker and can be run again.

Cancel a run when it is stuck and holds an execution slot that other pipelines are waiting for.

Messages

Troubleshooting

Run is greyed out. One of the three columns still shows an orange warning: no dataflow, an incomplete trigger, or an unmapped input or output location. Hover the warning for which.

The dataflow upload is refused as not valid for Cloud. The listed reasons name steps that cannot run in the cloud, most often a file read from a local path. Put the file behind a file location in Simplebim desktop and upload again.

After uploading a new dataflow the File Locations column turned orange. The new dataflow names a location the old one did not have. Open Map File Locations and map the new row.

Files appear in the trigger folder but nothing runs. Check in this order: the pipeline is Active and so is its workspace; the file name matches the Trigger File Filter; the file arrived after the pipeline was activated — files that were already waiting are only processed by pipelines that move trigger files to a handled folder, so run older files manually or drop them again; there is no .jobfailed marker next to the file; the connection is still authorized (Validate tells you).

Every run ends as Validation Failed. A folder is missing or a connection is not authorized. Validate shows which row fails; fix the folder or re-authorize the connection on the Connections page.

A run succeeded but is shown as Had Warnings. Either the dataflow reported warnings — the log lists them — or the trigger file could not be moved to the handled folder. The last lines of the log say which.

The same file ran twice. A file that is changed after its run counts as new and runs again — for example when it is saved again by a synchronisation client. Moving handled files to a handled folder avoids this.

Select Folder shows the connection's root, not my folder. The dialog is rooted at the chosen connection. If you need a folder outside it, choose another connection, or ask an administrator to change the connection's folder.

Open in … ↗ is greyed out. The folder was typed rather than picked, so its web address is unknown. Pick it once with Select Folder and save.

The Cancel button is not there although a run is stuck. Cancelling is an account administrator's action; ask one. A run also counts as stuck only while it is in progress — a finished run cannot be cancelled.

The ↑ vN badge will not go away. The pipeline is pinned to an older library version. Open the dataflow dialog and click Update to the latest version, or select the version you want with Select from library….