- Franklin Powers' Ortingo →
- Simple Data Toolkit - Tutorial - GUI - Batch Script
Simple Data Toolkit - Tutorial - GUI - Batch Script
July 12, 2022 at 5:09 PM UTC
This post has no media.
Franklin Powers
Let's suppose you want to create a task that loads a TSV file into a CSV file on a daily basis. This task is meant to be part of a Batch script, but you are unfamiliar with Batch and are unsure of how to write the code. Good news, you can use the Sample Data Toolkit UI to figure out this code for you as follows:
1) Go to the app page for the Simple Data Toolkit UI, at the time of this writing it is here: https://www.vis-software.com/#sdtk
2) Click Choose Files.
3) Find the file you want to convert.
4) Click Open.
5) Select the output type you want, for this tutorial we will select CSV.
6) Then for script, select Batch.
7) Then click Download Script.
You will now have a simple Batch script that uses STC (Simple Table Converter - A commandline utility that is part of the SDTK) to convert the file you selected to CSV. You can now either run this script or integrate it with another one.
If you need to install STC, you can install it via Chocolatey (https://chocolatey.org/) with the following command: choco install stc
The script will look something like this:
stc "ga_sample.csv" textonly "ga_sample.csv"
Check out these similar posts and Ortingos?
Simple Data Toolkit - Tutorial - GUI - JavaLet's suppose you want to create a task that loads a TSV file into a PSV file on a daily basis. This task is meant to be part of Java code, but you are unfamiliar with Java and are unsure of how to write the code. Good news, you can use the Sample Data Toolkit UI to figure out this code for you as follows: 1) Go to the app page for the Simple Data Toolkit UI, at the time of this writing it is here: https://www.vis-software.com/#sdtk 2) Click Choose Files. 3) Find the file you want to convert. 4) Click Open. 5) Select the output type you want, for this tutorial we will select PSV. 6) Then for script, select Java. 7) Then click Download Script. You will now have some Java code that uses SDTK to convert the file you selected to PSV. You can now either run this script or integrate it with another one. If you need to install SDTK for Java, you can do it by downloading the latest version from SourceForge and placing it in your class path. At the time of this writing, this can be found here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/sdtk.jar/download The code will look something like this: com.sdtk.table.Converter.start().readFile("complex.csv").tsv().textOnly().output().writeFile("complex.psv").psv().execute();Simple Data Toolkit - Tutorial - GUI - Python ScriptLet's suppose you want to create a task that loads a TSV file into a CSV file on a daily basis. This task is meant to be part of a Python script, but you are unfamiliar with Python and are unsure of how to write the code. Good news, you can use the Sample Data Toolkit UI to figure out this code for you as follows: 1) Go to the app page for the Simple Data Toolkit UI, at the time of this writing it is here: https://www.vis-software.com/#sdtk 2) Click Choose Files. 3) Find the file you want to convert. 4) Click Open. 5) Select the output type you want, for this tutorial we will select SQL. 6) Then for script, select Python. 7) Then click Download Script. You will now have a simple Python script that uses SDTK to convert the file you selected to CSV. You can now either run this script or integrate it with another one. If you need to install SDTK for Python, you can do it like so on the terminal/commandline prompt with the following command: pip install sdtk-visllc The script will look something like this: from sdtk import com_sdtk_table_Converter as sdtk sdtk.start().readFile("complex.csv").tsv().textOnly().output().writeFile("complex.csv").csv().execute();Simple Data Toolkit - Tutorial - PHP - Convert FileLet's suppose you want to create a task that loads a CSV file into an HTML file on a daily basis. This task is meant to be part of a PHP script, but you are unfamiliar with PHP and are unsure of how to write the code. Good news, you can use the Sample Data Toolkit UI to figure out this code for you as follows: 1) Go to the app page for the Simple Data Toolkit UI, at the time of this writing it is here: https://www.vis-software.com/#sdtk 2) Click Choose Files. 3) Find the file you want to convert. 4) Click Open. 5) Select the output type you want, for this tutorial we will select HTMLTable. 6) Then for script, select PHP. 7) Then click Download Script. You will now have a PHP script that uses SDTK to convert the file you selected to an HTML. You can now either run this script or integrate it with another one. If you need to install SDTK for PHP, you can do it by downloading the latest version from SourceForge, unzipping it, and placing it in your include path. At the time of this writing, this can be found here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/sdtk-php.zip/download The script will look something like this: readFile("ga_sample.csv")->csv()->textOnly()->output()->writeFile("ga_sample.htmltable")->htmlTable()->execute(); readfile("ga_sample.htmltable"); ?>Simple Data Toolkit - Tutorial - Windows - Scheduled TaskLet's suppose you want to create a task that loads a Pipe Delimited File into a database on a daily basis. The task is to run on Windows and you are using Snowflake for the database. You can do it with SDTK as follows: 1) Download SDTK. For this tutorial, we're going to use the latest version of the Windows Shell Script version as of this writing (0.1.2) which is here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/stc-wsh.js/download 2) Create a new text file called ConvertPSV.cmd 3) Open the file with a text editor (like Notepad or Visual Studio Code) Enter the following text: cscript stc-wsh.js clients.psv clients.sql createorreplace clients (For more info on STC and SDTK see the following URL: https://www.vis-software.com/#sdtk) (For more info on cscript see the following URL: https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/cscript) SET SNOWSQL_PWD=password snowsql -a myorganization-myaccount -u jsmith -f clients.sql -d database -s public -o quiet=true -o friendly=false (For more info on snowsql see the following URL: https://docs.snowflake.com/en/user-guide/snowsql-use.html) 4) Go to the Start Menu. 5) Search for Task Scheduler. (For more info on Task Scheduler see the following URL: https://docs.microsoft.com/en-us/windows/win32/taskschd/about-the-task-scheduler) 6) Open Task Scheduler. 7) Click Create Task. 8) For name enter: Convert PSV to SQL 9) Click Triggers Tab 10) Click New button 11) Select Daily 12) Start start time to 8:00 AM. 13) Click the OK button. 14) Click Actions tab. 15) Click New button 16) For Program/script click Browse. 17) Find the ConvertPSV.cmd script. 18) Click Open. 19) Set Start in to the directory that the ConvertPSV.cmd script is in. 20) Click the OK button. 21) Click the OK button. 22) To test, right click Convert PSV to SQL. 23) Select Run.Simple Data Toolkit - Tutorial - Mac OS - Scheduled TaskLet's suppose you want to create a task that loads a Pipe Delimited File into a database on a daily basis. The task is to run on Mac OS and you are using Snowflake for the database. You can do it with SDTK as follows: 1) Download SDTK. For this tutorial, we're going to use the latest version of the Python version as of this writing (0.1.2) which is here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/stc.py/download 2) Create a new text file called ConvertPSV.sh 3) Open the file with a text editor (like Text Edit) Enter the following text: #!/bin/sh python stc.py clients.psv clients.sql createorreplace clients (For more info on STC and SDTK see the following URL: https://www.vis-software.com/#sdtk) export SNOWSQL_PWD=password snowsql -a myorganization-myaccount -u jsmith -f clients.sql -d database -s public -o quiet=true -o friendly=false (For more info on snowsql see the following URL: https://docs.snowflake.com/en/user-guide/snowsql-use.html) 4) Go to the Terminal. 5) Edit the list of cron jobs with the following command: nano crontab -e 6) Add the following to a new line: 0 8 * * * cd ~ && ConvertPSV.sh 7) Press CTRL+O and CTRL+X to save and exit nano.Simple Data Toolkit - Tutorial - Linux - Scheduled TaskLet's suppose you want to create a task that loads a Pipe Delimited File into a database on a daily basis. The task is to run on Linux and you are using Snowflake for the database. You can do it with SDTK as follows: 1) Download SDTK. For this tutorial, we're going to use the latest version of the Python version as of this writing (0.1.2) which is here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/stc.py/download 2) Create a new text file called ConvertPSV.sh 3) Open the file with a text editor (like Text Edit) Enter the following text: #!/bin/sh python stc.py clients.psv clients.sql createorreplace clients (For more info on STC and SDTK see the following URL: https://www.vis-software.com/#sdtk) export SNOWSQL_PWD=password snowsql -a myorganization-myaccount -u jsmith -f clients.sql -d database -s public -o quiet=true -o friendly=false (For more info on snowsql see the following URL: https://docs.snowflake.com/en/user-guide/snowsql-use.html) 4) Go to the Terminal. 5) Edit the list of cron jobs with the following command: nano crontab -e 6) Add the following to a new line: 0 8 * * * cd ~ && ConvertPSV.sh 7) Press CTRL+O and CTRL+X to save and exit nano.Field Engine – Tutorial – Basic Scrollable Table with HTMLFrequently we need a table that a user can scroll through to present data for users. This is very easy with Field Engine and can be done with a few simple steps. We need to include the Field Engine files: <link href="FieldEngine.css" rel="stylesheet" /> <link href="FieldEngine-Defaults.css" rel="stylesheet" /> <script src="fe-browser.js"></script> Then we need to specify the CSS for the hex grid, we’re going to make our borders black and our hexes white: .field-hex .field_view_inner { background-color: black; } .field-hex .field_location { background-color: white; color: black; } Then we specify the scrollable table with a table and the class field-table <table class="field-table" columns="3" rows="3"> <tr><th>Col A</th><th>Col B</th></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> </table> And like that you have a scrollable table. Note that columns and rows specify the size of the view of the data.Field Engine – Tutorial – Basic Hex Grid with HTMLOn occassion, we need to display information in hex grids. With Field Engine, this is a relatively painless task. We need to include the Field Engine files: <link href="FieldEngine.css" rel="stylesheet" /> <link href="FieldEngine-Defaults.css" rel="stylesheet" /> <script src="fe-browser.js"></script> Then we need to specify the CSS for the hex grid, we’re going to make our borders black and our hexes white: .field-hex .field_view_inner { background-color: black; } .field-hex .field_location { background-color: white; color: black; } Then we specify the hex grid with a table and the class field-hex, like so: <table class="field-hex" columns="3" rows="3" borderWidth=”1px” borderColor=”black”> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> <tr><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td><td>123</td><td>456</td></tr> <tr><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td><td>789</td><td>012</td></tr> </table> And like magic, we have a hex grid.Field Engine – Tutorial – Basic Vertical Menu with HTMLAt times we need to display a vertical menu for users. This is very easy with Field Engine and can be done with a few simple steps. We need to include the Field Engine files: <link href="FieldEngine.css" rel="stylesheet" /> <link href="FieldEngine-Defaults.css" rel="stylesheet" /> <script src="fe-browser.js"></script> Then we need to specify the CSS for the hex grid, we’re going to make our borders black and our hexes white: .field-hex .field_view_inner { background-color: black; } .field-hex .field_location { background-color: white; color: black; } Then we specify the scrollable table with a ul tag and the class field-vertical-menu <ul class="field-vertical-menu"> <li>Menu</li> <li level="1" onclick="switchToTable">Table</li> <li level="1" onclick="switchToHex">Hex</li> </ul> And we'll have a fully functional vertical menu.Simple Data Toolkit - Python - Loading Images Through ChatGPTSimple Data Toolkit provides an API for passing images to ChatGPT and extracting related text. (At the time of this writing, the release of this is pending for complete support, but it is coming soon) To generate a LaTeX document based on a series of images (effectively doing OCR) we can do the following:
from sdtk import com_sdtk_api_ChatGPTAPI
import os
import fnmatch
import re
texData = ""
texBegin = "\n\\begin{document}\n"
texEnd = "\n\\end{document}"
header = ""
footer = ""
foundBody = False
# Get a list of all .png files in the current directory
png_files = [file for file in os.listdir('.') if fnmatch.fnmatch(file, '*.png')]
def result(text):
global texData
global header
global footer
global foundBody
text = text.replace("```latex", "").replace("```", "")
body_match = re.search(r'\\begin{document}(.*?)\\end{document}', text, re.DOTALL)
if body_match:
body_content = body_match.group(1).strip()
if len(texData) > 0:
texData = texData + "\n" + "\\newpage" + "\n"
else:
header_match = re.search(r'^(.*?)\\begin{document}', text, re.DOTALL)
header = header_match.group(1).strip()
footer_match = re.search(r'\\end{document}(.*)$', text, re.DOTALL)
footer = footer_match.group(1).strip()
texData = texData + body_content
foundBody = True
else:
foundBody = False
print(text)
# Loop over each .png file
for file_name in png_files:
foundBody = False
while foundBody == False:
com_sdtk_api_ChatGPTAPI.instance().query(result, "Can you generate a LaTeX document to represent the text in this image and format it correctly and return only the LaTeX code?", file_name)
with open("output.tex", "w+") as file:
# Write the string to the file
file.write(header + texBegin + texData + texEnd + footer)
Simple Data Toolkit - Tutorial - Git API - PythonSimple Data Toolkit provides an unofficial API for reading files, commits, branches, and repos from the Git API. (At the time of this writing, the release of this is pending for complete support, but it is coming soon)
To retrieve all repos a user has using Simple Data Toolkit, we can do the following:
from sdtk import com_sdtk_api_GitAPI
def printer(data, reader):
print(reader.toArrayOfNativeMaps(None))
com_sdtk_api_GitAPI.reposAPI().retrieveData({"owner": "Vis-LLC"}, printer)
To retrieve all branches a repo has using Simple Data Toolkit, we can do the following:
from sdtk import com_sdtk_api_GitAPI
def printer(data, reader):
print(reader.toArrayOfNativeMaps(None))
com_sdtk_api_GitAPI.branchesAPI().retrieveData({"owner": "Vis-LLC", "repo": "Simple-Data-Toolkit"}, printer)
To retrieve all the files in a branch using Simple Data Toolkit, we can do the following:
from sdtk import com_sdtk_api_GitAPI
def printer(data, reader):
print(reader.toArrayOfNativeMaps(None))
com_sdtk_api_GitAPI.filesAPI().retrieveData({"owner": "Vis-LLC", "repo": "Simple-Data-Toolkit", "branch": "main"}, printer)
To retrieve the data in a file using Simple Data Toolkit, we can do the following:
from sdtk import com_sdtk_api_GitAPI
def printerData(data, reader):
print(data)
com_sdtk_api_GitAPI.retrieveAPI().retrieveData({"owner": "Vis-LLC", "repo": "Simple-Data-Toolkit-UI", "branch": "main", "path": "index.html"}, printerData)
We can also login using a personal access token (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)
from sdtk import com_sdtk_api_GitAPI
def printerData(data, reader):
print(data)
com_sdtk_api_GitAPI.instance().setKey("Personal Access Token Here").retrieveAPI().retrieveData({"owner": "Vis-LLC", "repo": "Simple-Data-Toolkit-UI", "branch": "main", "path": "index.html"}, printerData)
Simple Data Toolkit - Tutorial - Ortingo API - PythonAt the time of this writing, Ortingo does not have an official API. Fortunately, Simple Data Toolkit provides an unofficial API for reading posts (at the time of this writing, the release of this is pending for complete support, but it is coming soon)
To retrieve all posts for a given user in Python, using Simple Data Toolkit, we can do the following:
from sdtk import com_sdtk_api_OrtingoAPI
def printer(data, reader):
print(reader.toArrayOfNativeMaps(None))
com_sdtk_api_OrtingoAPI.postsAPI().retrieveData({"owner": "60CQ59FN46SVQFXJ"}, printer)
Let's suppose we want only a list of titles for a given user, we can do this instead:
from sdtk import com_sdtk_api_OrtingoAPI
def printer(data, reader):
print(reader.filterColumnsOnly(["title"]).toArrayOfNativeMaps(None))
com_sdtk_api_OrtingoAPI.postsAPI().retrieveData({"owner": "60CQ59FN46SVQFXJ"}, printer)
We can also pull suggested content from Ortingo with the following, where the topics we are searching on are provided with the query parameter (in this case it's value is data):
from sdtk import com_sdtk_api_OrtingoAPI
def printerUrls(data, reader):
print(reader.filterColumnsOnly(["url"]).toArrayOfNativeMaps(None))
com_sdtk_api_OrtingoAPI.suggestionsAPI().retrieveData({"query": "data"}, printerUrls)
And finally, we can also pull comments attached to a post in Ortingo with the following, where the user is myself and the post is a test post I created:
from sdtk import com_sdtk_api_OrtingoAPI
def printerComments(data, reader):
print(reader.filterColumnsOnly(["commentDate", "post"]).toArrayOfNativeMaps(None))
com_sdtk_api_OrtingoAPI.commentsAPI().retrieveData({"owner": "60CQ59FN46SVQFXJ", "id": "test"}, printerComments)
The columns supported at the time of this writing are:
- id
- owner
- title
- subtitle
- post
- url
For comments the following columns are supported:
- id
- owner
- commentDate
- replyTo
- post
The workflow that works for you.
Drive results with a workflow so smooth, it doesn't feel like work. Move seamlessly between services that work in tandem, to discover untapped efficiency within your organization.
Ready to redefine your workflow?
Learn more about Ortingo
Any thoughts on Franklin's post?
To comment or reply, you need an Ortingo account.
Sign in or sign upHere's what Ortingoers think of Franklin's post.
There are no comments on this post.