================================================
Fault-tolerant LiveView Using Apache Kafka
================================================

Overview:

This sample demonstrates LiveView recovering from a service
interruption (a crash) using the open-source Apache Kafka as
the publishing bus. The LiveView Orders table is configured to use
persistence and it has a publisher interface with a Kafka subscriber.

The LiveView recovery protocol is documented in the LiveView Development Guide 
  
After LiveView service interruption and restart, LiveView recovers
from disk almost all of the rows that were in the Orders table at the
time of the interruption.

It is possible that some in-flight rows were not persisted before the
service interruption occurred. All messages that were published
while the server was unavailable must be recovered as well.

The LiveView publisher module in LVPublisher.sbapp
under fragments/publish/ references
the underlying bus implementation.

This implementation uses the StreamBase Kafka adapters, specifically the
Producer and Consumer. The implementation does not use the Administrator adapter.

Kafka documentation can be found in:

http://kafka.apache.org/documentation.html

Documentation for the StreamBase Kafka adapters can be found at:
  TIBCO StreamBase Documentation -> Adapters Guide -> StreamBase Embedded Adapters ->
      Kafka Producer Adapter, Kafka Consumer Adapter or Kafka Admin Adapter
  
You can use an existing Kafka server or, if you need a Kafka implementation, 
download the Kafka bundle from:

   http://kafka.apache.org/downloads.html 

This sample was developed using the Scala 2.12 based 0.10.2.1 version of Kafka, but 
other versions may work. Follow the instructions to install your server.

To run the server, use the following instructions. The 
files required to run on either Windows or Linux are supplied. The two operating systems 
require slightly different commands. 

Start three terminal windows and cd to the directory where you
installed Kafka. The installation includes both the Kafka server and 
a ZooKeeper. The ZooKeeper must be started first:

In the first terminal, type:
  (Linux)$ bin/zookeeper-server-start.sh config/zookeeper.properties 
(Windows)> bin/windows/zookeeper-server-start config/zookeeper.properties

Now in the second terminal, start the Kakfa server:
  (Linux)$ bin/kafka-server-start.sh config/server.properties
(Windows)> bin/windows/kafka-server-start config/server.properties

If you have not already done so, in the third terminal create a new topic for this item:
  (Linux)$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Orders
(Windows)> bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Orders

The topic must be created once each time the Kafka server and 
ZooKeeper are reset, as described below. 

To confirm the topic is created:
  (Linux)$ bin/kafka-topics.sh --list --zookeeper localhost:2181
(Windows)> bin/windows/kafka-topics --list --zookeeper localhost:2181

In the sample base directory, build everything and start the LiveView Server:

 $ mvn install -DskipTests=true
 $ epadmin install node nodename=A.kafka nodedirectory=/tmp/nodes application=`ls deploy_apps/peer_deploy/target/*.zip`

Start the publisher: 

 $ epadmin install node nodename=B.kafka_pub nodedirectory=/tmp/nodes application=`ls deploy_apps/publish_deploy/target/*.zip`
  
Start both services:
  
 $ epadmin servicename=kafka start node
 $ epadmin servicename=kafka_pub start node

Note that you need at least 8GB of memory for the log demo and at least 12GB for the log and peer demo.

Connect a LiveView Desktop to the LiveView Server running on port
10087 and confirm you data is being published. Shut down the
publisher and wait for the Desktop output to stop:

 $ epadmin servicename=kafka_pub stop node

Then shut down the LiveView Server:

 $ epadmin servicename=kafka_pub stop node

To launch the peer recovery sample, first ensure the log sample is running. Then start the peer:

 $ epadmin install node nodename=C.kafka2 nodedirectory=/tmp/nodes application=`ls deploy_apps/peer_deploy/target/*.zip`
 $ epadmin servicename=kafka2 start node
 

To restart the broker and ZooKeeper from scratch and delete any saved
messages, terminate both servers using the following commands:
  (Linux)$ bin/kafka-server-stop.sh (Linux)
  (Linux)$ bin/zookeeper-server-stop.sh (Linux)
  
(Windows)> bin/windows/kafka-server-stop
(Windows)> bin/windows/zookeeper-server-stop

and delete the configured log directories (/tmp/zookeeper
and /tmp/kafka-logs by default). Then restart the servers.

The recovery data is stored in /tmp/lvout/data and needs to be removed between runs

If you are running on Linux, there is a test bash script provided that
can be used to demonstrate LiveView recovery.
 
The script starts the publisher and then crashes the LiveView server
(3 times by default). It then compares the data in the LiveView table
with the data that was published. Despite having many service
interruptions, the data should match exactly. The script performs this
comparison. To run this script:

./liveview-bounce.sh

There is a corresponding script that crashes the peer setup:

./liveview-bounce-peers.sh

Both of them use the /tmp/lvout and /tmp/lvout2 to store compiled SBARs. These 
folders need to be removed for a fresh start to the scripts.

Under Windows, stop the lv-server (not the MesssagePublisher) using
your preferred method (Task manager or an epadmin remove with the 
installpath to simulate a crash) and then restart it as above. 

Modifying this component for your use: 

Note that Kafka example does not require any changes to the 
data that is currently placed on the bus. Also, adding
a PublisherID or Publisher serial number to the payload is not 
required as the sample uses the topic and offset for those purposes.

The MessagePublisher is straightforward. The main point to remember is 
to modify the Broker, partition and topic as appropriate. In 
the sample, they are set up as parameters. The table publisher must 
match these values; the KafkaPublisher also has these values as parameters.

Note that the interface of record for the your LiveView table resides in
Orders.sbint. The named schemas in this interface are
imported by the LiveView lvconf files, which in turn are the basis of
the generated interfaces in lv-interfaces. Start with Orders.sbint when 
making table schema changes for your tables.