This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Data Input

Operators in the Data Input category

1: Arrow File Scan
2: CSV File Scan
3: CSVOld File Scan
4: File Lister
5: File Scan
6: File Scan From Input
7: JSONL File Scan
8: Text Input

Home > Data Input

Operators

Operator	Description
Arrow File Scan	Scan data from an Arrow file
CSV File Scan	Scan data from a CSV file
CSVOld File Scan	Scan data from a CSVOld file
File Lister	Select a dataset version and output one filename tuple per file
File Scan	Scan data from a file
File Scan From Input	Scan data from file paths provided by input tuples
JSONL File Scan	Scan data from a JSONL file
Text Input	Source data from manually inputted text

Total: 8 operators

1 - Arrow File Scan

Scan data from an Arrow file

Home > Data Input

Input Properties

Property	Requirement	Type	Default	Description
File	✓	String	-
Limit		Integer	-	Max output count
Offset		Integer	-	Starting point of output

Output Ports

Port	Mode
0	Set Snapshot

2 - CSV File Scan

Scan data from a CSV file

Home > Data Input

Input Properties

Property	Requirement	Type	Default	Description
File	✓	String	-
File Encoding	✓	UTF_8, UTF_16, US_ASCII	UTF_8	Decoding charset to use on input
Limit		Integer	-	Max output count
Offset		Integer	-	Starting point of output
Delimiter		String	,	Delimiter to separate each line into fields
Header		Boolean	true	Whether the CSV file contains a header line

Output Ports

Port	Mode
0	Set Snapshot

3 - CSVOld File Scan

Scan data from a CSVOld file

Home > Data Input

Input Properties

Property	Requirement	Type	Default	Description
File	✓	String	-
File Encoding	✓	UTF_8, UTF_16, US_ASCII	UTF_8	Decoding charset to use on input
Limit		Integer	-	Max output count
Offset		Integer	-	Starting point of output
Delimiter		String	,	Delimiter to separate each line into fields
Header		Boolean	true	Whether the CSV file contains a header line

Output Ports

Port	Mode
0	Set Snapshot

4 - File Lister

Select a dataset version and output one filename tuple per file

Home > Data Input

Input Properties

Property	Requirement	Type	Default	Description
Dataset	✓	String	-

Output Ports

Port	Mode
0	Set Snapshot

5 - File Scan

Scan data from a file

Home > Data Input

Input Properties

Property	Requirement	Type	Default
File	✓	String	-
Encoding	✓	UTF_8, UTF_16, US_ASCII	UTF_8
Extract		Boolean	false
↳ Include Filename		Boolean	false
Attribute Type	✓	string, single string, integer, long, double, boolean, timestamp, binary, large binary	string
Attribute Name	✓	String	line
Limit		Integer	-
Offset		Integer	-

Output Ports

Port	Mode
0	Set Snapshot

6 - File Scan From Input

Scan data from file paths provided by input tuples

Home > Data Input

Input Properties

Property	Requirement	Type	Default
Encoding	✓	UTF_8, UTF_16, US_ASCII	UTF_8
Extract		Boolean	false
Include Filename		Boolean	false
Attribute Type	✓	string, single string, integer, long, double, boolean, timestamp, binary, large binary	string
Attribute Name	✓	String	line
Limit		Integer	-
Offset		Integer	-

Output Ports

Port	Mode
0	Set Snapshot

7 - JSONL File Scan

Scan data from a JSONL file

Home > Data Input

Input Properties

Property	Requirement	Type	Default	Description
File	✓	String	-
File Encoding	✓	UTF_8, UTF_16, US_ASCII	UTF_8	Decoding charset to use on input
Limit		Integer	-	Max output count
Offset		Integer	-	Starting point of output
Flatten	✓	Boolean	false	Flatten nested objects and arrays

Output Ports

Port	Mode
0	Set Snapshot

8 - Text Input

Source data from manually inputted text

Home > Data Input

Input Properties

Property	Requirement	Type	Default
Text	✓	String	-
Attribute Type	✓	string, single string, integer, long, double, boolean, timestamp, binary, large binary	string
Attribute Name	✓	String	line
Limit		Integer	-
Offset		Integer	-

Output Ports

Port	Mode
0	Set Snapshot