Simple Data Toolkit - Tutorial - Linux - Scheduled Task

July 12, 2022 at 2:49 PM PST

This post has no media.

Let's suppose you want to create a task that loads a Pipe Delimited File into a database on a daily basis. The task is to run on Linux and you are using Snowflake for the database. You can do it with SDTK as follows:

1) Download SDTK. For this tutorial, we're going to use the latest version of the Python version as of this writing (0.1.2) which is here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/stc.py/download

2) Create a new text file called ConvertPSV.sh

3) Open the file with a text editor (like Text Edit)
Enter the following text:
#!/bin/sh
python stc.py clients.psv clients.sql createorreplace clients
(For more info on STC and SDTK see the following URL: https://www.vis-software.com/#sdtk)

export SNOWSQL_PWD=password
snowsql -a myorganization-myaccount -u jsmith -f clients.sql -d database -s public -o quiet=true -o friendly=false
(For more info on snowsql see the following URL: https://docs.snowflake.com/en/user-guide/snowsql-use.html)

4) Go to the Terminal.

5) Edit the list of cron jobs with the following command:
nano crontab -e

6) Add the following to a new line:
0 8 * * * cd ~ && ConvertPSV.sh

7) Press CTRL+O and CTRL+X to save and exit nano.

Any thoughts on Franklin's post?

To comment or reply, you need an Ortingo account.

Here's what Ortingoers think of Franklin's post.

There are no comments on this post.

Check out these similar posts and Ortingos?

Simple Data Toolkit - Tutorial - Mac OS - Scheduled TaskLet's suppose you want to create a task that loads a Pipe Delimited File into a database on a daily basis. The task is to run on Mac OS and you are using Snowflake for the database. You can do it with SDTK as follows: 1) Download SDTK. For this tutorial, we're going to use the latest version of the Python version as of this writing (0.1.2) which is here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/stc.py/download 2) Create a new text file called ConvertPSV.sh 3) Open the file with a text editor (like Text Edit) Enter the following text: #!/bin/sh python stc.py clients.psv clients.sql createorreplace clients (For more info on STC and SDTK see the following URL: https://www.vis-software.com/#sdtk) export SNOWSQL_PWD=password snowsql -a myorganization-myaccount -u jsmith -f clients.sql -d database -s public -o quiet=true -o friendly=false (For more info on snowsql see the following URL: https://docs.snowflake.com/en/user-guide/snowsql-use.html) 4) Go to the Terminal. 5) Edit the list of cron jobs with the following command: nano crontab -e 6) Add the following to a new line: 0 8 * * * cd ~ && ConvertPSV.sh 7) Press CTRL+O and CTRL+X to save and exit nano.Simple Data Toolkit - Tutorial - Windows - Scheduled TaskLet's suppose you want to create a task that loads a Pipe Delimited File into a database on a daily basis. The task is to run on Windows and you are using Snowflake for the database. You can do it with SDTK as follows: 1) Download SDTK. For this tutorial, we're going to use the latest version of the Windows Shell Script version as of this writing (0.1.2) which is here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/stc-wsh.js/download 2) Create a new text file called ConvertPSV.cmd 3) Open the file with a text editor (like Notepad or Visual Studio Code) Enter the following text: cscript stc-wsh.js clients.psv clients.sql createorreplace clients (For more info on STC and SDTK see the following URL: https://www.vis-software.com/#sdtk) (For more info on cscript see the following URL: https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/cscript) SET SNOWSQL_PWD=password snowsql -a myorganization-myaccount -u jsmith -f clients.sql -d database -s public -o quiet=true -o friendly=false (For more info on snowsql see the following URL: https://docs.snowflake.com/en/user-guide/snowsql-use.html) 4) Go to the Start Menu. 5) Search for Task Scheduler. (For more info on Task Scheduler see the following URL: https://docs.microsoft.com/en-us/windows/win32/taskschd/about-the-task-scheduler) 6) Open Task Scheduler. 7) Click Create Task. 8) For name enter: Convert PSV to SQL 9) Click Triggers Tab 10) Click New button 11) Select Daily 12) Start start time to 8:00 AM. 13) Click the OK button. 14) Click Actions tab. 15) Click New button 16) For Program/script click Browse. 17) Find the ConvertPSV.cmd script. 18) Click Open. 19) Set Start in to the directory that the ConvertPSV.cmd script is in. 20) Click the OK button. 21) Click the OK button. 22) To test, right click Convert PSV to SQL. 23) Select Run.Simple Data Toolkit - Tutorial - GUI - JavaLet's suppose you want to create a task that loads a TSV file into a PSV file on a daily basis. This task is meant to be part of Java code, but you are unfamiliar with Java and are unsure of how to write the code. Good news, you can use the Sample Data Toolkit UI to figure out this code for you as follows: 1) Go to the app page for the Simple Data Toolkit UI, at the time of this writing it is here: https://www.vis-software.com/#sdtk 2) Click Choose Files. 3) Find the file you want to convert. 4) Click Open. 5) Select the output type you want, for this tutorial we will select PSV. 6) Then for script, select Java. 7) Then click Download Script. You will now have some Java code that uses SDTK to convert the file you selected to PSV. You can now either run this script or integrate it with another one. If you need to install SDTK for Java, you can do it by downloading the latest version from SourceForge and placing it in your class path. At the time of this writing, this can be found here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/sdtk.jar/download The code will look something like this: com.sdtk.table.Converter.start().readFile("complex.csv").tsv().textOnly().output().writeFile("complex.psv").psv().execute();Simple Data Toolkit - Tutorial - PHP - Convert FileLet's suppose you want to create a task that loads a CSV file into an HTML file on a daily basis. This task is meant to be part of a PHP script, but you are unfamiliar with PHP and are unsure of how to write the code. Good news, you can use the Sample Data Toolkit UI to figure out this code for you as follows: 1) Go to the app page for the Simple Data Toolkit UI, at the time of this writing it is here: https://www.vis-software.com/#sdtk 2) Click Choose Files. 3) Find the file you want to convert. 4) Click Open. 5) Select the output type you want, for this tutorial we will select HTMLTable. 6) Then for script, select PHP. 7) Then click Download Script. You will now have a PHP script that uses SDTK to convert the file you selected to an HTML. You can now either run this script or integrate it with another one. If you need to install SDTK for PHP, you can do it by downloading the latest version from SourceForge, unzipping it, and placing it in your include path. At the time of this writing, this can be found here: https://sourceforge.net/projects/simple-data-toolkit/files/0.1.3/sdtk-php.zip/download The script will look something like this: readFile("ga_sample.csv")->csv()->textOnly()->output()->writeFile("ga_sample.htmltable")->htmlTable()->execute(); readfile("ga_sample.htmltable"); ?>Simple Data Toolkit - Tutorial - GUI - Python ScriptLet's suppose you want to create a task that loads a TSV file into a CSV file on a daily basis. This task is meant to be part of a Python script, but you are unfamiliar with Python and are unsure of how to write the code. Good news, you can use the Sample Data Toolkit UI to figure out this code for you as follows: 1) Go to the app page for the Simple Data Toolkit UI, at the time of this writing it is here: https://www.vis-software.com/#sdtk 2) Click Choose Files. 3) Find the file you want to convert. 4) Click Open. 5) Select the output type you want, for this tutorial we will select SQL. 6) Then for script, select Python. 7) Then click Download Script. You will now have a simple Python script that uses SDTK to convert the file you selected to CSV. You can now either run this script or integrate it with another one. If you need to install SDTK for Python, you can do it like so on the terminal/commandline prompt with the following command: pip install sdtk-visllc The script will look something like this: from sdtk import com_sdtk_table_Converter as sdtk sdtk.start().readFile("complex.csv").tsv().textOnly().output().writeFile("complex.csv").csv().execute();Simple Data Toolkit - Tutorial - GUI - Batch ScriptLet's suppose you want to create a task that loads a TSV file into a CSV file on a daily basis. This task is meant to be part of a Batch script, but you are unfamiliar with Batch and are unsure of how to write the code. Good news, you can use the Sample Data Toolkit UI to figure out this code for you as follows: 1) Go to the app page for the Simple Data Toolkit UI, at the time of this writing it is here: https://www.vis-software.com/#sdtk 2) Click Choose Files. 3) Find the file you want to convert. 4) Click Open. 5) Select the output type you want, for this tutorial we will select CSV. 6) Then for script, select Batch. 7) Then click Download Script. You will now have a simple Batch script that uses STC (Simple Table Converter - A commandline utility that is part of the SDTK) to convert the file you selected to CSV. You can now either run this script or integrate it with another one. If you need to install STC, you can install it via Chocolatey (https://chocolatey.org/) with the following command: choco install stc The script will look something like this: stc "ga_sample.csv" textonly "ga_sample.csv"Simple Data Toolkit - Tutorial - Git API - PythonSimple Data Toolkit provides an unofficial API for reading files, commits, branches, and repos from the Git API. (At the time of this writing, the release of this is pending for complete support, but it is coming soon) To retrieve all repos a user has using Simple Data Toolkit, we can do the following:

	from sdtk import com_sdtk_api_GitAPI

	def printer(data, reader):
	    print(reader.toArrayOfNativeMaps(None))

	com_sdtk_api_GitAPI.reposAPI().retrieveData({"owner": "Vis-LLC"}, printer)

To retrieve all branches a repo has using Simple Data Toolkit, we can do the following:

	from sdtk import com_sdtk_api_GitAPI

	def printer(data, reader):
	    print(reader.toArrayOfNativeMaps(None))

	com_sdtk_api_GitAPI.branchesAPI().retrieveData({"owner": "Vis-LLC", "repo": "Simple-Data-Toolkit"}, printer)

To retrieve all the files in a branch using Simple Data Toolkit, we can do the following:

	from sdtk import com_sdtk_api_GitAPI

	def printer(data, reader):
	    print(reader.toArrayOfNativeMaps(None))

	com_sdtk_api_GitAPI.filesAPI().retrieveData({"owner": "Vis-LLC", "repo": "Simple-Data-Toolkit", "branch": "main"}, printer)

To retrieve the data in a file using Simple Data Toolkit, we can do the following:

	from sdtk import com_sdtk_api_GitAPI

	def printerData(data, reader):
	    print(data)

	com_sdtk_api_GitAPI.retrieveAPI().retrieveData({"owner": "Vis-LLC", "repo": "Simple-Data-Toolkit-UI", "branch": "main", "path": "index.html"}, printerData)

We can also login using a personal access token (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)

	from sdtk import com_sdtk_api_GitAPI

	def printerData(data, reader):
	    print(data)

	com_sdtk_api_GitAPI.instance().setKey("Personal Access Token Here").retrieveAPI().retrieveData({"owner": "Vis-LLC", "repo": "Simple-Data-Toolkit-UI", "branch": "main", "path": "index.html"}, printerData)

Simple Data Toolkit - Tutorial - Ortingo API - PythonAt the time of this writing, Ortingo does not have an official API. Fortunately, Simple Data Toolkit provides an unofficial API for reading posts (at the time of this writing, the release of this is pending for complete support, but it is coming soon) To retrieve all posts for a given user in Python, using Simple Data Toolkit, we can do the following:

from sdtk import com_sdtk_api_OrtingoAPI


def printer(data, reader):
    print(reader.toArrayOfNativeMaps(None))

com_sdtk_api_OrtingoAPI.postsAPI().retrieveData({"owner": "60CQ59FN46SVQFXJ"}, printer)

Let's suppose we want only a list of titles for a given user, we can do this instead:

from sdtk import com_sdtk_api_OrtingoAPI


def printer(data, reader):
    print(reader.filterColumnsOnly(["title"]).toArrayOfNativeMaps(None))

com_sdtk_api_OrtingoAPI.postsAPI().retrieveData({"owner": "60CQ59FN46SVQFXJ"}, printer)

We can also pull suggested content from Ortingo with the following, where the topics we are searching on are provided with the query parameter (in this case it's value is data):

from sdtk import com_sdtk_api_OrtingoAPI

def printerUrls(data, reader):
    print(reader.filterColumnsOnly(["url"]).toArrayOfNativeMaps(None))

com_sdtk_api_OrtingoAPI.suggestionsAPI().retrieveData({"query": "data"}, printerUrls)

And finally, we can also pull comments attached to a post in Ortingo with the following, where the user is myself and the post is a test post I created:

from sdtk import com_sdtk_api_OrtingoAPI

def printerComments(data, reader):
    print(reader.filterColumnsOnly(["commentDate", "post"]).toArrayOfNativeMaps(None))

com_sdtk_api_OrtingoAPI.commentsAPI().retrieveData({"owner": "60CQ59FN46SVQFXJ", "id": "test"}, printerComments)

from sdtk import com_sdtk_api_IEEEAPI,com_sdtk_calendar_IEEEEventFormat

def printer(data, reader):
    ieee = com_sdtk_calendar_IEEEEventFormat.instance
    for event in reader.toArrayOfNativeMaps(None):
        ci = ieee.read(event)
        print(ci.summary)


com_sdtk_api_IEEEAPI.eventsAPI().retrieveData({"limit": "2"}, printer)

We can search using the following parameters: - limit - The limit to the number of events to return - start - The start datetime to search - end - The end datetime to search The columns supported at the time of this writing are: - created-at mapped to created - start-time mapped to start - end-time mapped to end - title mapped to summary - uid mapped to uid Simple Data Toolkit - Tutorial - BitTorrent API - PythonSimple Data Toolkit provides an API for retrieving transactions from BitTorrent. (At the time of this writing, the release of this is pending for complete support, but it is coming soon) To retrieve all transactions in Python, using Simple Data Toolkit, we can do the following:

from sdtk import com_sdtk_api_BitTorrentAPI

def printer(data, reader):
    print(reader.toArrayOfNativeMaps(None))

com_sdtk_api_BitTorrentAPI.transactionsAPI().retrieveData({}, printer)

We can also convert the transactions, to SDTK's internal event/calendar format with the BitTorrentFormat class like so:

from sdtk import com_sdtk_api_BitTorrentAPI,com_sdtk_calendar_BitTorrentFormat

def printerEvents(data, reader):
    etherScan = com_sdtk_calendar_BitTorrentFormat.instance()
    for event in reader.toArrayOfNativeMaps(None):
        ci = etherScan.read(event)
        print(ci)

com_sdtk_api_BitTorrentAPI.transactionsAPI().retrieveData({}, printerEvents)

We can also search by address, startTimestamp, and endTimestamp.

Simple Data Toolkit - Tutorial - Linux - Scheduled Task

July 12, 2022 at 2:49 PM PST

This post has no media.

Any thoughts on Franklin's post?

Here's what Ortingoers think of Franklin's post.

Check out these similar posts and Ortingos?

Made with ❤️ + 💻 + ☕️ in Silicon Valley

© 2024 Ortingo

Report Abuse

Privacy Policy

User Agreement

API (Unofficial)

For Investors