Spark MLlib Operator Sample

Introduction

This sample shows basic usage of the Spark MLlib model evaluator. The scenario uses an audit data dataset from the PMML examples at http://dmg.org/pmml/pmml_examples/index.html.

The EventFlow module in this sample shows an example of runtime binomial classification. The input events contain some demographics data about bank customers. The events leave the flow enriched by the Target information computed by the model.

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

  • From the top-level menu, select File>Load StreamBase Sample.

  • Enter Spark to narrow the list of options.

  • Select Spark from the Large Data Storage and Analysis category.

  • Click OK.

StreamBase Studio creates a single project for the Spark operator samples in your current Studio workspace.

Running the Spark Sample in StreamBase Studio

Run this sample in Studio as follows:

  1. In the Project Explorer view, open the sample you just loaded.

    If you see red marks on a project folder, wait a moment for the project to load its features.

    If the red marks do not resolve themselves after a minute, select the project, right-click, and select Maven>Update Project from the context menu.

  2. Open the src/main/eventflow/packageName folder.

  3. Open the audit.sbapp file and click the Run button. This opens the SB Test/Debug perspective and starts the fragment.

  4. In the Feed Simulations view, select the audit.sbfs and click Run.

  5. In the Output Streams view, select All Output Streams and view the data emitted by the EventFlow fragment. It starts with tuples similar to the following:

    Target=0, Age=38.0, Employment=Private, Education=College, Marital=Unmarried, 
      Occupation=Service, Income=81838.0, Gender=Female, Deductions=0.0, Hours=72.0
    Target=0, Age=35.0, Employment=Private, Education=Associate, Marital=Absent, 
      Occupation=Transport, Income=72099.0, Gender=Male, Deductions=0.0, Hours=30.0
    Target=0, Age=32.0, Employment=Private, Education=HSgrad, Marital=Divorced, 
      Occupation=Clerical, Income=154676.74, Gender=Male, Deductions=0.0, Hours=40.0
    Target=0, Age=45.0, Employment=Private, Education=Bachelor, Marital=Married, 
      Occupation=Repair, Income=27743.82, Gender=Male, Deductions=0.0, Hours=55.0
    Target=0, Age=60.0, Employment=Private, Education=College, Marital=Married, 
      Occupation=Executive, Income=7568.23, Gender=Male, Deductions=0.0, Hours=40.0
    Target=0, Age=74.0, Employment=Private, Education=HSgrad, Marital=Married, 
      Occupation=Service, Income=33144.4, Gender=Male, Deductions=0.0, Hours=30.0
    Target=1, Age=43.0, Employment=Private, Education=Bachelor, Marital=Married, 
      Occupation=Executive, Income=43391.17, Gender=Male, Deductions=0.0, Hours=50.0
    Target=0, Age=35.0, Employment=Private, Education=Yr12, Marital=Married, 
      Occupation=Machinist, Income=59906.65, Gender=Male, Deductions=0.0, Hours=40.0
    Target=0, Age=25.0, Employment=Private, Education=Associate, Marital=Divorced, 
      Occupation=Clerical, Income=126888.91, Gender=Female, Deductions=0.0, Hours=40.0
    Target=0, Age=22.0, Employment=Private, Education=HSgrad, Marital=Absent, 
      Occupation=Sales, Income=52466.49, Gender=Female, Deductions=0.0, Hours=37.0
    Target=0, Age=48.0, Employment=Private, Education=College, Marital=Divorced, 
      Occupation=Service, Income=291416.11, Gender=Female, Deductions=0.0, Hours=35.0
  6. When done, press F9 or click the Terminate EventFlow Fragment button.

Running the Spark Artifact Management Sample in StreamBase Studio

The artifact management sample demonstrates how you can use a TIBCO Artifact Management Server to feed data into the Spark MLlib operator to load new version of models at runtime.

The TIBCO Artifact Management Server is a separately installed product. This sample assumes that you have a running Artifact Management Server (AMS) that is accessible by StreamBase. Refer to the product documentation for more information regarding AMS.

This sample also requires that you do the following:

  1. Create a project folder in the Artifact Management Server called sample.

  2. Navigate to sample/models-spark from your StreamBase installation directory and upload the audit.zip file to the sample project folder. This file is a compressed version of the audit folder. Note that the sample/models-spark directory in StreamBase is a different folder from the AMS sample project folder.

Run this sample in Studio as follows:

  1. In the Project Explorer view, open the sample you just loaded.

    If you see red marks on a project folder, wait a moment for the project to load its features.

    If the red marks do not resolve themselves after a minute, select the project, right-click, and select Maven>Update Project from the context menu.

  2. Open the src/main/eventflow/packageName folder.

  3. Open the artifactmanagement.sbapp file and click the Run button. This opens the SB Test/Debug perspective and starts the module.

  4. The application will try to pull the artifacts from AMS. Note that the Required On Startup option is selected for all operators, which means all the artifacts must be loaded from the AMS system before the application will run. If all artifacts cannot be loaded within a specific period of time, the system will fail with an artifact timeout exception.

  5. In the Feed Simulations view, select the audit.sbfs and click Run.

  6. In the Output Streams view, select All Output Streams and view the data emitted by the StreamBase application. It starts with tuples similar to the following:

    Target=0, Age=38.0, Employment=Private, Education=College, Marital=Unmarried, 
      Occupation=Service, Income=81838.0, Gender=Female, Deductions=0.0, Hours=72.0
    Target=0, Age=35.0, Employment=Private, Education=Associate, Marital=Absent, 
      Occupation=Transport, Income=72099.0, Gender=Male, Deductions=0.0, Hours=30.0
    Target=0, Age=32.0, Employment=Private, Education=HSgrad, Marital=Divorced, 
      Occupation=Clerical, Income=154676.74, Gender=Male, Deductions=0.0, Hours=40.0
    Target=0, Age=45.0, Employment=Private, Education=Bachelor, Marital=Married, 
      Occupation=Repair, Income=27743.82, Gender=Male, Deductions=0.0, Hours=55.0
    Target=0, Age=60.0, Employment=Private, Education=College, Marital=Married, 
      Occupation=Executive, Income=7568.23, Gender=Male, Deductions=0.0, Hours=40.0
    Target=0, Age=74.0, Employment=Private, Education=HSgrad, Marital=Married, 
      Occupation=Service, Income=33144.4, Gender=Male, Deductions=0.0, Hours=30.0
    Target=1, Age=43.0, Employment=Private, Education=Bachelor, Marital=Married, 
      Occupation=Executive, Income=43391.17, Gender=Male, Deductions=0.0, Hours=50.0
    Target=0, Age=35.0, Employment=Private, Education=Yr12, Marital=Married, 
      Occupation=Machinist, Income=59906.65, Gender=Male, Deductions=0.0, Hours=40.0
    Target=0, Age=25.0, Employment=Private, Education=Associate, Marital=Divorced, 
      Occupation=Clerical, Income=126888.91, Gender=Female, Deductions=0.0, Hours=40.0
    Target=0, Age=22.0, Employment=Private, Education=HSgrad, Marital=Absent, 
      Occupation=Sales, Income=52466.49, Gender=Female, Deductions=0.0, Hours=37.0
    Target=0, Age=48.0, Employment=Private, Education=College, Marital=Divorced, 
      Occupation=Service, Income=291416.11, Gender=Female, Deductions=0.0, Hours=35.0
  7. From AMS, deploy the model again to the running artifactmanagement.sbapp application using the following properties:

    1. Target Type: StreamBase Service Name

    2. StreamBase Service Name The node name that appears in the Studio Clusters view for this application

  8. When done, press F9 or click the Terminate EventFlow Fragment button.

Sample Location

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is:

studio-workspace/sample_operator_models_Spark

See Default Installation Directories for the default location of studio-workspace on your system.