Contents
These samples illustrate how to access the Amazon S3 file system using the TIBCO StreamBase® File Writer for Apache Hadoop Distributed File System (HDFS) and TIBCO StreamBase® File Reader for Apache Hadoop Distributed File System (HDFS).
In StreamBase Studio, import this sample with the following steps:
-
From the top menu, select
→ . -
Type
s3
to narrow the list of options. -
Select
from the category. -
Click
.
StreamBase Studio creates a single project containing the sample files.
Open the sample application, FileBasic.sbapp
or FileAdvanced.sbapp
, select the Parameters tab, and edit the value to represent:
-
Your current S3 bucket
-
Where you would like to store the sample data
You must also open the sample S3 configuration file, sample_s3a.conf
, and enter your security access keys for your S3 file system. See this link for other ways to authenticate with S3: Authenticating with S3.
-
In the Package Explorer, double-click to open the
FileBasic.sbapp
application. Make sure the application is the currently active tab in the EventFlow Editor. -
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
In the Manual Output view, switch the
Stream
toData
, then enter a string value such astest
. -
Click
to send a data tuple to be written to the file. Repeat for as many lines as you wish. -
In the Application Output view, observe tuples emitted on the
Status
output streams indicating actions performed on the file. -
In the Manual Output view, switch the
Stream
toWriteControl
, then enterClose
into theCommand
field. -
Click
to send a control tuple, which closes the current file for writing. -
In the Manual Output view, switch the
Stream
toReadControl
, then click to send a control tuple, which reads the default file. -
Press F9 or click the Stop Running Application button.
-
This demo has now created a file in your S3 file system called
sample/Sample.txt
containing the lines of data you submitted.
-
In the Package Explorer, double-click to open the
FileAdvanced.sbapp
application. Make sure the application is the currently active tab in the EventFlow Editor. -
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
In the Application Output view, observe tuples emitted on the
Status
output streams indicating actions performed to the files. -
Press F9 or click the Stop Running Application button.
-
This demo has now created multiple files in your S3 file system:
-
sample/Sample.gz
— This file is a GZip compressed file created from theSampleIn.txt
file. -
sample/Sample.gz2
— This file is a BZip2 compressed file created from theSampleIn.txt
file. -
sample/Sample.zip
— This file is a Zip compressed file created from theSampleIn.txt
file. -
sample/SampleOut.txt
— This file is an uncompressed file created from theSampleIn.txt
file.
-
When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.
Important
Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.
Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is:
studio-workspace
/sample_adapter_embedded_hdfsS3
See Default Installation Directories for the default location of studio-workspace
on your system.