A guide to the Dataflow Library of Simplebim Cloud: what it is, how it is organised into an account library and workspace libraries, how dataflows and their versions are published, browsed, compared and downloaded, who may see and publish what, and how pipelines use library dataflows.
The library is reached from three places — the Library tab of a workspace, the Library tab of the Account page and the Library page itself — all described here. How a pipeline is pointed at a library dataflow is part of Using Pipelines; publishing to and getting from the library in Simplebim desktop is described in Using the Simplebim Cloud Dialogs.
Contents
- What the Dataflow Library is
- Account library and workspace libraries
- Dataflows and versions
- Where you find the library
- Browsing an area
- Publishing to the library
- Downloading a version
- Comparing versions
- Permissions
- Library dataflows in pipelines
- Enabling the library for an account
- Messages
- Troubleshooting
What the Dataflow Library is
Without the library, a dataflow lives only inside the pipeline it was uploaded to. Ten pipelines that run the same dataflow hold ten copies, each updated by hand, and nobody can tell afterwards which copy ran when.
The Dataflow Library is a central, versioned store of dataflows for an account. A dataflow in the library has a name, a numbered version history that can never be rewritten, a record of who published each version and why, and its own permissions. Pipelines refer to a library dataflow instead of holding a private copy: several pipelines can share one dataflow, a new version can reach all of them at once, or each can stay pinned to the version it was tested with.
The library is an option that Simplebim enables per account. In an account without it, the Library tabs and pages do not exist and dataflows are uploaded straight into pipelines. In an account with it, both routes remain: a dataflow can still be uploaded directly into a pipeline when sharing is not wanted. The Account page's Details tab shows Library — Enabled since … when the account has it.
Account library and workspace libraries
The library is divided into areas. Every account with the library has these built-in areas, created and named automatically:
| Area | How many | Name | Meant for |
|---|---|---|---|
| Account library | One per account. | The account's name. | Dataflows shared across the whole account — the company's standard checks, exports and conversions. |
| Workspace library | One per workspace. | The workspace's name, shown as Account → Workspace on the Library page. | Dataflows that belong to one project: variants tuned to that project's models, one-off conversions, work in progress. |
The two kinds differ in who sees them. A workspace library is visible automatically to everyone who has access to that workspace, and everyone who may edit the workspace may publish into its library. The account library is closed by default: only account administrators see it until they grant access to individual members. See Permissions.
They do not differ in what they can hold or how pipelines use them: a pipeline in any workspace may use a dataflow from the account library or from its own workspace's library — or, when granted, from another workspace's library. A good rule of thumb is to develop and test a dataflow in the workspace library where it is needed first, and to publish it into the account library once it is meant to be reused.
When a workspace is renamed, its library follows; when a workspace is deleted, its library goes with it.
Dataflows and versions
A library dataflow is identified by its name within an area — two dataflows in one area cannot share a name. It carries a description and holds versions numbered v1, v2, v3… in the order they were published.
A version is created by publishing and never changes afterwards. It records:
| Recorded | What it is |
|---|---|
| Version | The running number; the newest carries the latest badge. |
| Comment | What the publisher wrote about this version — what changed, why. |
| Cost | The tool level of the most demanding tool in the dataflow, as Level N, or Undefined. It decides how many of the account's tokens a run consumes. |
| Published / Published by | When and by whom. |
| warnings badge | Remarks found when the dataflow was examined at publish time. Hover the badge to read them; they do not prevent use. |
Publishing the same content again does not create a version; see When nothing new is created. Versions cannot be deleted, so a pipeline pinned to an old version keeps working for as long as anyone wants it to.
A dataflow that is no longer meant to be used can be archived. An archived dataflow shows the Archived badge in the list, is not offered when a pipeline chooses a dataflow, and cannot receive new versions under its name — a fresh upload with that name creates a new dataflow. Its versions stay readable and pipelines still pointing at it keep running.
Where you find the library
| Place | What it shows |
|---|---|
| Workspaces › a workspace › Library tab | That workspace's library. The tab exists only in accounts with the library. |
| Account › Library tab | The account library. |
| Library page (administrators, profile menu › Admin › Library) | All areas of the account — account and workspace libraries — as a list with Area, Kind, Description and a Permissions button per area. Click an area's name to open it; the breadcrumb Library / area leads back. |
All three places open the same area browser, described next. A member who has no access to an area sees You do not have access to … Ask an account administrator to grant you access. instead of its content.
Browsing an area
The dataflow list
An opened area lists its dataflows:
| Column | What it shows |
|---|---|
| Dataflow | The name, with an Archived badge when archived. Click it to open the version list. |
| Latest Version | The newest version number. |
| Updated | When the newest version was published. |
| Description | The dataflow's description. |
| Versions | Opens the version list. |
| Permissions | Opens the dataflow's own permissions. See The Permissions dialog. |
Above the list, Upload New Dataflow publishes a file into the area (see Uploading in the browser) and Reload refreshes the list — for example after a colleague has published from Simplebim desktop. An empty area says This area has no dataflows yet. Upload a dataflow file, or publish one from the Simplebim desktop client.
The version list
Opening a dataflow shows its versions, newest first, with the columns listed under Dataflows and versions and a Download button per version. The breadcrumb Dataflows / name leads back to the list. The buttons above the list are Upload New Version, Compare… (enabled once there are two versions) and Reload.
Publishing to the library
Publishing needs publish rights in the target area or on the target dataflow; see Permissions.
From Simplebim desktop
The usual way. Build and test the dataflow in Simplebim desktop, then choose Publish Current to Cloud on the Cloud menu of the Dataflow palette and use the dialog's Library tab: pick the Area, give the Name and a Comment, and click Publish. Areas you may only read from are marked (read-only). The dialog is described in Using the Simplebim Cloud Dialogs. Reload the list in Simplebim Cloud to see the result.
Uploading in the browser
Upload New Dataflow in an area, or Upload New Version on a dataflow, opens the upload dialog. Drop a .sbdf file — a dataflow saved by Simplebim desktop — onto the drop zone or click it to browse. The file is examined first and the result is shown before anything is stored:
- a green The dataflow is valid for Cloud. with the dataflow's Name and Cost, followed by what Save will do: Saving creates the new dataflow '…'. or Saving publishes a new version of '…'.;
- a red The dataflow is not valid for Cloud and cannot be saved: with the reasons — typically a file read from a local disk rather than a file location, or a tool the cloud does not have. Fix the dataflow in Simplebim desktop and upload again;
- possibly a yellow Warnings list, which is stored with the version but does not prevent saving.
Write a Comment and click Save. The message The dataflow was published to the library. confirms it. Cancel discards the staged file.
When uploading into an area, the dataflow's own name — the name it has in Simplebim desktop — decides where it goes: a name that no dataflow in the area has yet creates a new dataflow with v1; a name that exists adds the next version to that dataflow. Upload New Version on a dataflow always adds to that dataflow, whatever the file is called.
When nothing new is created
A version is identified by its content. Publishing a file whose content and cost are identical to the newest version creates nothing: the library keeps its current version and the newest number stays. Simplebim desktop says so — Identical content is already published as version N — no new version was created.; the browser upload reports a successful publish but the version list does not grow. Change the dataflow before publishing again.
Downloading a version
Download in the version list saves the version as a .sbdf file named dataflow_vN.sbdf, ready to be opened in Simplebim desktop with Open on the Dataflow palette. Use it to inspect or edit an older version, or to bring a colleague's dataflow to your own computer. Simplebim desktop can also fetch a version directly with Get as Current from Cloud on the Cloud menu.
Downloading changes nothing in the library.
Comparing versions
Compare… in the version list opens Compare Versions. Choose the From and To versions — the two newest are preselected — and the dialog shows what changed between them as the dataflow's underlying definition: added lines in green with a +, removed lines in red with a −, unchanged parts collapsed to · · · with a few lines of context around each change. The counts of added and removed lines are shown next to the selectors. The selected versions have identical content. means exactly that.
The comparison is meant for answering "what did version 4 change?" — it shows the step configurations as text, not a graphical dataflow.
Permissions
Who sees and publishes what
Library access is closed by default and opened by grants. Two rights exist: Read — see the area or dataflow, download versions, use them in pipelines and fetch them in Simplebim desktop — and Publish — Read plus publishing new versions and dataflows and managing permissions.
| Who | Account library | Workspace library |
|---|---|---|
| Account administrator | Read and Publish, always. | Read and Publish, always. |
| Member with access to the workspace (Users tab of the workspace) | Nothing, unless granted below. | Read; Publish when the workspace grant includes Write. |
| Member with a grant on the area | As granted. | As granted, in addition to the above. |
| Member with a grant on one dataflow | That dataflow only, as granted. The area becomes visible so the dataflow can be found. | Same. |
| Anyone else | Nothing. | Nothing. |
So a project team sees its own workspace library without any setup, and the account library is shared deliberately — either area-wide, or one dataflow at a time.
The Permissions dialog
Permissions on an area (Library page) or on a dataflow (dataflow list) opens the same editor as the workspace's Users tab: a table of grants with User, Read — always ticked — and Publish. Add New Permission adds a row; pick the member — account administrators are not offered, they have everything already — tick Publish if wanted, and Save. The trash icon removes a grant; Cancel discards unsaved changes. Below, Permissions through Account Roles lists the account administrators for reference.
Account administrators and members who themselves hold Publish on the area or dataflow may edit its grants; others see the list read-only.
Library dataflows in pipelines
A pipeline in a library account can take its dataflow from the library instead of an upload. In the pipeline's dataflow dialog, Select from library… asks for the area, the dataflow and the version:
- Latest — the pipeline follows the dataflow. Whenever a run starts, the newest version at that moment is used, and the run records which version it was.
- Specific version — the pipeline is pinned to that version until somebody changes it. When a newer version exists the pipeline shows a blue ↑ vN badge, and Update to the latest version in the dataflow dialog moves the pin.
The dataflow dialog shows Library dataflow and Version — v3 (pinned to v3) or v3 (follows latest), with — v4 available when there is a newer one. Uploading a file into such a pipeline detaches it from the library: the upload becomes the pipeline's private dataflow and library versions no longer reach it. Selecting from the library again re-attaches it.
Whether to follow latest or pin is a question of how the dataflow is maintained. Pipelines that should pick up improvements automatically follow latest; pipelines in a project that must not change mid-way stay pinned and are updated on purpose. Details of the dialog are in Using Pipelines.
Enabling the library for an account
The library is enabled per account by Simplebim. When it is enabled for an account that already has pipelines, every pipeline's current dataflow is imported into the library of the pipeline's workspace as version v1 of a dataflow with the same name, and the pipeline is pinned to that version. Nothing changes in how the pipelines run; from then on new versions can be published to the library and the pipelines updated when wanted. Dataflows that were uploaded into several pipelines become separate library dataflows, one per pipeline, and can be consolidated afterwards.
Should the library later be disabled for the account, the Library tabs disappear but nothing is deleted: the areas, dataflows, versions and grants are kept and come back if it is enabled again, and pipelines keep the dataflow they last received.
Messages
| Message | Meaning |
|---|---|
| This account has not been upgraded to the Dataflow Library yet. | The account does not have the library. |
| You do not have access to any dataflow library areas yet. Ask an account administrator to grant you access. | No area is visible to you; see Permissions. |
| You do not have access to this workspace's library area. / …the account's library area. | The same, for one tab. |
| This area has no dataflows yet. Upload a dataflow file, or publish one from the Simplebim desktop client. | The area is empty. |
| The dataflow is valid for Cloud. / The dataflow is not valid for Cloud and cannot be saved: | Result of examining the uploaded file. |
| Saving creates the new dataflow '…'. / Saving publishes a new version of '…'. | What Save will do with the staged file. |
| The selected file is too large to be a dataflow file. | The file is not a .sbdf dataflow file. |
| The dataflow was published to the library. | Published; reload the list if it does not show yet. |
| The selected versions have identical content. | Nothing differs between the compared versions. |
| Failed to download the dataflow content. Please try again. | The download did not complete; retry. |
| All account users already have permissions. | Every member who could be granted access already has a row. |
Troubleshooting
There is no Library tab. The account does not have the Dataflow Library. Dataflows are uploaded straight into pipelines instead; see Using Pipelines.
I see my workspace's library but not the account library. The account library is closed by default. Ask an account administrator for a grant on the account area, or on the dataflow you need.
I published from Simplebim desktop but the list does not show it. Click Reload; the browser does not refresh by itself. If it is still missing, check that you published into the area you are looking at — the desktop dialog lists workspace areas and the account area side by side.
Publishing did not create a new version. The content and cost are identical to the newest version. Change the dataflow before publishing.
Upload New Dataflow added a version to an existing dataflow instead of creating a new one. The file's dataflow name matched an existing dataflow in the area. Rename the dataflow in Simplebim desktop and save the file again if it is meant to be a new one.
The dataflow I need is not offered when selecting a dataflow for a pipeline. Either you lack Read on it, or it is archived. Archived dataflows stay readable in the list but are not offered to pipelines.
A pipeline still runs the old version although a new one was published. The pipeline is pinned. Open its dataflow dialog and click Update to the latest version, or select the dataflow again with Latest.
Compare… is greyed out. The dataflow has only one version; there is nothing to compare with.
I cannot edit the permissions. Editing needs account administrator rights or a Publish grant of your own on that area or dataflow.
Comments
0 comments
Please sign in to leave a comment.