Methodologies

Getting data into your SimplyWork instance is a critical first step in getting started, and an key component in the ongoing continuous integration with external systems such as payroll. The import process can be divided into the following steps:

1. Fetch or receive the external data
2. Parse and translate as needed to match the internal requirements
3. Update the internal SimplyWork datasources

The following table discusses the options for receiving data into SimplyWork:

Method	Description	Timing
File upload	You upload or drag-n-drop a static file into the import app; this file is usually created in Excel and exported to CSV, or is a report you run in another system and save to CSV	On demand, upload whenever you have a new file
SFTP retrieve	A file is fetched via SFTP on a schedule; the retrieved file is placed on the SimplyWork SFTP site (for which you'll have a guest account), or some other SFTP site hosted by others; the file is usually CSV or a fixed width text file	On a schedule, e.g. daily at 5am
API GET	Data is retrieved from another site via an API, usually delivered in a JSON, XML or CSV format	On a schedule, e.g. daily at 5am
API POST	Data is posted into SimplyWork by an external source, also in a JSON, XML or CSV format	Whenever the external system sends it; the import apps simply waits for data to arrive

Import Types

The diagrams below demonstrates how external data is exchanged via SFTP between a "system of record" payroll partner, with the internal datasources within SimplyWork.

File Upload

The graphic belows demonstrates the manual file upload processes; in this case the import only occurs in response to the manual uploading of a file:

In this use-cases, the Data Import app is waiting for you to upload a file.

1. A file is uploaded to the import app
2. The rules and other tools within the Data Import translate the data into the necessary format
3. Data is saved in SimplyWork

Back to Imort Types

SFTP

The graphic belows demonstrates the scheduled SFTP process:

The data is fetched on a schedule you manage; you and the other party agree on a timeline so-as to keep the data reasonably current

1. The external partner places the file on sftp.simplywork.com
2. You schedule a Data Import app to retrieve the file each morning at a specified time
3. The rules and other tools within the Data Import translate the data into the necessary format
4. Data is saved in SimplyWork

Back to Import Types

API GET

The graphic belows demonstrates the scheduled API GET process:

In this case, like SFTP, data is fetched on a schedule you manage but is retrieved directly from the source; it could be considered "fresher"

1. You schedule a Data Import app to retrieve the file each morning at a specified time via API
2. The rules and other tools within the Data Import translate the data into the necessary format
3. Data is saved in SimplyWork

Back to Import Types

API POST

The graphic belows demonstrates the API POST process; in this case the import only occurs in response to the external event:

In this use-case, the Data Import app is waiting for some external service to post data.

1. An external service (payroll partner, another app in your domain) posts data to the SimplyWork API Queue
2. The rules and other tools within the Data Import translate the data into the necessary format
3. Data is saved in SimplyWork

Back to Imort Types

Data Formats

Regardless of the transpost methodology chosen, the actual data exchange can be in any of the following formats. Here we take a look at what a list of Departments for your employees might look like.

CSV

Departments.csv

Code,Description,Status
ADM,Administration,Active
IT,Information Technology,Active
MAIN,Maintenance,Active
DLVRY,Delivery,Inactive
SAL,Sales & Marketing,Active

JSON

Departments.json

[{"Code":"ADM", "Description":"Administration", "Status":"Active"}, 
{"Code":"IT", "Description":"Information Technology", "Status":"Active"}, 
{"Code":"MAIN", "Description":"Maintenance", "Status":"Active"}, 
{"Code":"DLVRY", "Description":"Delivery", "Status":"Inactive"}, 
{"Code":"SAL", "Description":"Sales & Marketing", "Status":"Active"}]

XML

Departments.xml

<Departments>
  <Department>
    <Code>ADM</Code>
    <Description>Administration</Description>
    <Status>Active</Status>
  </Department>
  <Department>
    <Code>IT</Code>
    <Description>Information Technology</Description>
    <Status>Active</Status>
  </Department>
  <Department>
    <Code>MAIN</Code>
    <Description>Maintenance</Description>
    <Status>Active</Status>
  </Department>
  <Department>
    <Code>DLVRY</Code>
    <Description>Delivery</Description>
    <Status>Inactive</Status>
  </Department>
  <Department>
    <Code>SAL</Code>
    <Description>Sales & Marketing</Description>
    <Status>Active</Status>
  </Department>      
</Departments>

Back to Data Formats

SOAP

The SOAP format is also supported, though we'll cover that in another section.

Filtering

Once uploaded or received, the source data is previewed and optional filters can be applied to limit which rows are imported.

The above shows a filter in the form of Type == (JobCode) which is interprted as the "Type" column contains the static value "Job Code"

Filtering is useful when the incoming file or data stream contains records which you don't wish to import, or might belong in a different table within SimplyWork. This allows importing the same file multiple times, each pass extracting those rows which are applicable to the given import target.

Mapping

The heart of the import process is Mapping where the incoming source values are mapped to a target value in the target data store. Some mappings, such as a name, may be directly mappable as-is. Many others will require some manipulation, such as a translation (e.g. "true" -> "1") or formatting (e.g. "4/11/2022" -> "2022-04-11") or a lookup (e.g. the location "101" -> lookup("Locations", "Code") to perform an internal lookup).

The above mapping shows Job Codes being processed; Code and Description are copied directly to the value of the same name. Status is translated from "Inactive" to "1"

Mappings also provide a "before" and "after" preview of what the data looks like before the mapping interaction vs after. The "after" version is what will be stored in the target.

Scheduling

For imports which fetch files or data, you can optionally schedule to run periodically, such as daily or weekly in order to automate the process.

Logs

The final section of imports is a job log that shows all past scheduled imports. This provides an audit trail summary of when an import ran and high-level stats summarizing the results.

Next we'll walk through the steps of creating a new import app using the Import Wizard.