Visual editor tool info


Request
The SaaS ETL platform had a Visual editor that allowed building a data transformation flow diagram with its tools. Each tool required a documentation page for it, the one that would provide both technical info on how exactly the tool transforms data and a couple of examples for it.

Solution
"Data merging tools" pages with "Aggregate by Values" page (for that very tool) among others.
For each tool I wrote a dedicated page that had the tool's description, parameters to set, JSON-code templates and a couple of "original stats → the tool set a certain way → the resulting stats" style examples. All such pages had the same structure and features, like a JSON-code template copy button, that made it easier to work with the tools in Visual editor.

Platform and tools used
Docs site run on a static site generator; Git, VSCode, Markdown, Mac OS image editors.


 

☝ Tech name: transform/keepTitle
     Composable

MM Visual editor diagram node:

Description

Allows saving the entities in JC.it database, through the MM integration, by their titles instead of ids from the data source integration (title values get copied into the originalId subcolumn).

Parameters

  • targetField [required] - entities key

JSON-code templates

  • MM Visual editor 
    [
  {
    "targetField": "_entitiesKey_"
  }
]
  • Full component code 
    {
  "name": "transform/keepTitle",
  "params": [
    {
      "targetField": "_entitiesKey_"
    }
  ],
  "comment": "any information"
}

- EXAMPLES -

1. General example

There are two data sources: facebook and tiktok.
In the facebook data flow, the country entity had been created by using the New Entities component, whereas the tiktok had the country entity available to be collected from source initially.
So both data flows have the same meaning and titles entities before the merge point and thus there will be duplicated statistics after that point.
To avoid that, this component will combine both entities data into one country entity after the data flows merge point.

  1. The original stats

    By data_source:

    data_source country country.originalId view
    facebook UK 94975234 5
    facebook AU 23087089 3
    tiktok UK 2356083 10

    By country:

    country view
    UK 5
    AU 3
    UK 10

    Instead of collapsing into a single row, the UK stats got separated.

  2. The component’s JSON-code

    To prevent such separation, the Aggregate by Values component can be used:

     [
   {
     "targetField": "country"
   }
 ]

  3. The resulting stats
    By data_source

    data_source country country.originalId view
    facebook UK UK 5
    facebook AU AU 3
    tiktok UK UK 10

    By country

    country view
    UK 15
    AU 3

2. Detailed flow example

Single data source

Let’s take a look at how the single source data is saved in JC.it Core by data source module integration and loaded into the MM integration:

  1. Original source data - saved in the Core, .id’s are assigned to all the entities in entities group (here - campaign)

    campaign.title campaign.id campaign.originalId
    camp_01 10 1000
    camp_02 20 2000

  2. In MM without Aggregate by Values: campaign.id gets rewritten creating new entities for this integration; campaign.originalId copied from source integration’s campaign.id

    data_source campaign.title campaign.id campaign.originalId
    source_01 camp_01 30 10
    source_01 camp_02 40 20

  3. In MM with Aggregate by Values: campaign.id gets rewritten creating new entities for this integration; campaign.originalId copied from source integration’s campaign.title

    data_source campaign.title campaign.id campaign.originalId
    source_01 camp_01 30 camp_01
    source_01 camp_02 40 camp_02

Multiple data sources

What about processing two+ data sources stats? Or, for some reports, the entities that have been added using the New Entities component and the entities that have being collected from data source, both have same titles. In such case, the statistics within the entities group will get separated: both entities with same title will be displayed due to having different originalIds. This component will merge such same titles entities into one by copying their (same) titles into their originalIds.

  1. Data source 1 integration

    campaign.title campaign.id campaign.originalId views
    camp_01 10 1000 10
    camp_02 20 2000 10

  2. Data source 2 integration

    campaign.title campaign.id campaign.originalId views
    camp_01 30 10000 5
    camp_02 40 20000 5

  3. In MM without Aggregate by Values: campaign.id gets rewritten creating new entities; campaign.originalId copied from source integration’s campaign.id

    Viewing by data_source

    data_source campaign.title campaign.id campaign.originalId views
    source_01 camp_01 50 10 10
    source_01 camp_02 60 20 10
    source_02 camp_01 70 30 5
    source_02 camp_02 80 40 5

    Using source integration’s ids (10-20 and 30-40) as originalIds here and matching the latter to source integrations’ ids, MM finds the entities one by one and loads their titles and metrics. Totally new ids (50-80) are assigned to this MM integration’s new entities.

    Viewing by campaign.title

    campaign.title campaign.id campaign.originalId views
    camp_01 50 10 10
    camp_02 60 20 10
    camp_01 70 30 5
    camp_02 80 40 5

    All the rows describe completely different entities. Hence both campaigns data got separated and stats is incorrect in all the rows.

  4. In MM with Aggregate by Values: campaign.id gets rewritten creating new entities; campaign.originalId copied from source integration’s campaign.title

    Viewing by data_source

    data_source campaign.title campaign.id campaign.originalId views
    source_01 camp_01 50 camp_01 10
    source_01 camp_02 60 camp_02 10
    source_02 camp_01 50 camp_01 5
    source_02 camp_02 60 camp_02 5

    Using source integration’s title (camp_01 and camp_02) as originalId here and matching the latter to source integrations’ title, MM: finds the entity, loads its metrics, assigns totally new ids (50-60) to this MM integration’s new entities OR same ones - if such an entity (with same originalId and title) has already been created within this integration.

    Viewing by campaign.title

    campaign.title campaign.id campaign.originalId views
    camp_01 50 camp_01 15
    camp_02 60 camp_02 15

    Thanks to this component, rows that contain same campaigns data have become rows of same entities - with sameoriginalId and title - hence the stats could sum up, becoming correct for both campaigns.