Install Oracle NoSQL Enterprise

The easiest way to learn Oracle NoSQL is to download the Oracle Big Data Lite VirtualBox image (here). That image contains most components used in Oracle Big Data technology stack including Cloudera Manager. There are quite a few steps, and you will need at least 100GB free space on your VirtualBox host. The steps on the Oracle web site are very clear. One reminder is to allocation minimum 16GB physical memory and minimum 2 cores to the VirtualBox guest. The image has everything in standalone configuration as it has only 1 node. From learning perspective that is a great beginning.

A better way to learn Oracle NoSQL, however, is to install Oracle NoSQL Enterprise on a cluster. This practice, we will go over the steps needed for Oracle NoSQL Enterprise edition installation and some basic operations.

Oracle NoSQL can be downloaded from OTN. You can get both the binaries and examples.

Step 1: Prepare software and system

Unzip and move the software to /opt/app

  • [hadoop@nnode1 Downloads]$ tar xvfz kv-ee-4.3.11.tar.gz
  • [hadoop@nnode1 Downloads]$ tar xvfz kv-examples-4.3.11.tar.gz
  • [hadoop@nnode1 Downloads]$ sudo mv kv-4.3.11 /opt/app

Open up needed ports through firewall. Login as root on each participating node as root to execute following:

firewall-cmd --permanent --zone=internal --add-port=30000-49999/tcp
firewall-cmd --permanent --zone=internal --add-port=1000-5999/tcp
firewall-cmd --reload

Disable swap on each participating node. Oracle NoSQL is greedy in memory usage. It will by default allocate about 90% total memory. If swap is enabled, it will assume a bigger total memory size, and will end up with many swapping activities. Swapping slows down database performance. To disable swapping, edit /etc/fstab and comment out all swap file systems. After that, reboot the node, and then check with command “free -m”. You should see 0 under total swap.

Step 2: Create single node KV store to verify the environment

Use KVLite, you can quickly create a KV store and start using NoSQL right away. This is a good way to verify that your environment is working fine even if you are about to configure the enterprise KV store. In one terminal, go to the folder /opt/app/kv-4.3.11, and then:

[hadoop@nnode1 kv-4.3.11]$ java -jar lib/kvstore.jar kvlite
Generated password for user admin: 6]9-~vkVKbZ6
User login file: ./kvroot/security/user.security
Created new kvlite store with args:
-root ./kvroot -store kvstore -host nnode1 -port 5000 -secure-config enable

In another terminal, go to the folder /opt/app/kv-4.3.11, and then ping the KV store.

hadoop@nnode1 kv-4.3.11]$ java -jar lib/kvstore.jar ping -host nnode1 -port 5000 -security kvroot/security/user.security
Pinging components of store kvstore based upon topology sequence #14
10 partitions and 1 storage nodes
Time: 2017-03-12 21:06:03 UTC Version: 12.1.4.3.11
Shard Status: healthy:1 writable-degraded:0 read-only:0 offline:0
Admin Status: healthy
Zone [name=KVLite id=zn1 type=PRIMARY allowArbiters=false] RN Status: online:1 offline:0
Storage Node [sn1] on nnode1:5000 Zone: [name=KVLite id=zn1 type=PRIMARY allowArbiters=false] Status: RUNNING Ver: 12cR1.4.3.11 2017-02-17 06:52:50 UTC Build id: 0e3ebe7568a0
Admin [admin1] Status: RUNNING,MASTER
Rep Node [rg1-rn1] Status: RUNNING,MASTER sequenceNumber:54 haPort:5006

Note the zone name is KVLite, and status is RUNNING. Also you can compile and run the sample applications.

javac -cp examples:lib/kvclient.jar examples/hello/HelloBigDataWorld.java
java -Doracle.kv.security=kvroot/security/user.security -cp examples:lib/kvclient.jar hello.HelloBigDataWorld

Step 3: Replicate the installation onto other nodes

Add following to /home/hadoop/.bashrc.

export KVHOME=/opt/app/kv-4.3.11
export KVROOT=/opt/data/kvroot
export PATH=$PATH:$KVHOME/bin
export CLASSPATH=$CLASSPATH:$KVHOME/lib/*

Change to root by “su -” on nnode1, and then:

for i in `cat /home/hadoop/mysites | grep -v nnode1`; do
rsync -avzhe ssh /opt/app/kv-4.3.11 $i:/opt/app
rsync -avzhe ssh /home/hadoop/.bashrc $i:/home/hadoop
done

Verify files are replicated properly on all nodes. Next, create the KVROOT. Leave the /opt/app/kv-4.3.11/kvroot untouched. That is the KVROOT for the KVLite Zone. We will not use that one, but rather create an official KVROOT in the data folder. In a production system, the /opt/data folder can be mounted to a storage designated for data, so that you can have data and executables (including configurations and logs) in separate places.

[root@nnode1 ~]# mkdir /opt/data/kvroot
[root@nnode1 ~]# chown hadoop:dev /opt/data/kvroot'
[root@nnode1 ~]# ssh nnode2 'mkdir /opt/data/kvroot; chown hadoop:dev /opt/data/kvroot'
[root@nnode1 ~]# ssh dnode1 'mkdir /opt/data/kvroot; chown hadoop:dev /opt/data/kvroot'

Step 3: Create KV configuration on each node

Next, run makebootconfig command on each node to generate configuration files.

[hadoop@nnode1 ~]$ java -jar $KVHOME/lib/kvstore.jar makebootconfig -root /opt/data/kvroot -port 5000 -host nnode1 -harange 5010,5025
[hadoop@nnode1 ~]$ ssh nnode2 'java -jar ${KVHOME}/lib/kvstore.jar makebootconfig -root /opt/data/kvroot -port 5000 -host nnode2 -harange 5010,5025'
[hadoop@nnode1 ~]$ ssh dnode1 'java -jar ${KVHOME}/lib/kvstore.jar makebootconfig -root /opt/data/kvroot -port 5000 -host dnode1 -harange 5010,5025'

Here we use 5010 to 5025 as HA ports for storage node agent inter-communication.

Next, we will create the security files. The command will generate a list of files.

[hadoop@nnode1 kv-4.3.11]$ java -jar ./lib/kvstore.jar securityconfig config create -root /opt/data/kvroot -kspwd xxxxxx
Created files
/opt/data/kvroot/security/store.keys
/opt/data/kvroot/security/store.trust
/opt/data/kvroot/security/store.wallet/cwallet.sso
/opt/data/kvroot/security/security.xml
/opt/data/kvroot/security/client.trust
/opt/data/kvroot/security/client.security
Created

These security files are to be replicated to all nodes. Note, avoid generating the same security files on all nodes.

for i in `cat /home/hadoop/mysites | grep -v nnode1`; do
rsync -avzhe ssh /opt/data/kvroot/security $i:/opt/data/kvroot/
done

Step 4: Start storage node agent (SNA) on each node

The command starts a SNA on a node. Note for now, the SNAs are not connected yet. After each command, use “jps -m” command to verify that the KV store process is running.

nohup java -Xmx1024m -Xms256m -jar ${KVHOME}/lib/kvstore.jar start -root ${KVROOT} > /opt/app/kv-4.3.11/kvstore.log 2>&1 &
ssh nnode2 ‘nohup java -Xmx1024m -Xms256m -jar ${KVHOME}/lib/kvstore.jar start -root ${KVROOT} > /opt/app/kv-4.3.11/kvstore.log 2>&1 &’
ssh dnode1 ‘nohup java -Xmx1024m -Xms256m -jar ${KVHOME}/lib/kvstore.jar start -root ${KVROOT} > /opt/app/kv-4.3.11/kvstore.log 2>&1 &’

Also, trying to connect to each SNA. for now you can use random user name and password, as we have not defined users yet. The following message indicate the SNA is running properly. Note the message about “5000 is not registered” is fine. That basically says the SNA’s have not formed a cluster.

[hadoop@nnode1 kv-4.3.11]$ java -jar ${KVHOME}/lib/kvstore.jar ping -host nnode1 -port 5000 -security ${KVROOT}/security/client.security
Login as:root
root's password:
SNA at hostname: nnode1, registry port: 5000 is not registered.
No further information is available
Can't find store topology: Could not contact any RepNode at: [nnode1:5000]

Step 5: Deploy the first administration process on nnode1

Deploying administration processes is a multiple step process. We need to deploy 1 administration process on the first node. After finish admin user creation, we can add additional administration processes. Normally we should have a least 3 administration processes in a cluster.

First, start CLI by the runadmin command. You will see the “kv->” prompt.

java -Xmx256m -Xms256m -jar ${KVHOME}/lib/kvstore.jar runadmin -port 5000 -host nnode1 -security ${KVROOT}/security/client.security

Once in CLI, we will issue following 4 commands. Note after each command, I added a “show topology” command so you can see how the topology got evolved.

  • name the KV store to kvestore
  • name the zone as zone1
  • register the storage node
  • create an administration service

kv-> configure -name kvestore
kv-> show topology
store=kvestore numPartitions=0 sequence=0

kv-> plan deploy-zone -name zone1 -rf 1 -wait
Executed plan 1, waiting for completion…
Plan 1 ended successfully
kv-> show topology
store=kvestore numPartitions=0 sequence=1
zn: id=zn1 name=zone1 repFactor=1 type=PRIMARY allowArbiters=false

kv-> plan deploy-sn -zn zn1 -host nnode1 -port 5000 -wait
Executed plan 2, waiting for completion…
Plan 2 ended successfully
kv-> show topology
store=kvestore numPartitions=0 sequence=2
zn: id=zn1 name=zone1 repFactor=1 type=PRIMARY allowArbiters=false
sn=[sn1] zn:[id=zn1 name=zone1] nnode1:5000 capacity=1 RUNNING

kv-> plan deploy-admin -sn sn1 -wait
Executed plan 3, waiting for completion…
Plan 3 ended successfully

Next, we will create a storage node pool, and add the first storage node to the pool. We name the pool as pool1 here.

kv-> pool create -name pool1
kv-> pool join -name pool1 -sn sn1
kv-> show pools
AllStorageNodes: sn1 zn:[id=zn1 name=zone1]
pool1: sn1 zn:[id=zn1 name=zone1]

Step 6: Deploy rest administration processes

Next, we add the rest nodes to the pool. Note the topology change after adding all storage nodes.

kv-> plan deploy-sn -zn zn1 -host nnode2 -port 5000 -wait
kv-> pool join -name pool1 -sn sn2
kv-> plan deploy-sn -zn zn1 -host dnode1 -port 5000 -wait
kv-> pool join -name pool1 -sn sn3

Now we can create the rest administration services. Note here we create 3 administration services on 3 out of 4 nodes.

kv-> plan deploy-admin -sn sn2 -wait
kv-> plan deploy-admin -sn sn3 -wait

*** one problem I had when deploy admin service on sn3 is network time issue. The 3rd node when out of sync with other nodes. Had to reinstall NTP service. And it went through.

Step 7: Create replication nodes

Here we created total 40 partitions. This will automatically create a number of replication nodes.

kv-> topology create -name topo1 -pool pool1 -partitions 9
kv-> plan deploy-topology -name topo1 -wait

At the end, you should get a topology like this:

Troubleshooting

  • show topology: display a summary of everything configured in the KVROOT
  • show admins: display all admin services and their statues
  • start admin service: plan start-service -service admin3 -wait
  • Typical jps output on a SNA with admin service
    [hadoop@dnode1 ~]$ jps -m
    3040 ManagedService -root /opt/data/kvroot/kvestore/sn3 -secdir /opt/data/kvroot/security -store kvestore -class Admin -service admin3
    1619 ManagedService -root /opt/data/kvroot/kvestore/sn3 -secdir /opt/data/kvroot/security -store kvestore -class RepNode -service rg3-rn1
    1540 kvstore.jar start -root /opt/data/kvroot
    3099 Jps -m