Data LakeData Lake Administration Guide

Download Exabeam_[product]_[build_version].sxb file from Exabeam Community that you want to install. Transfer the downloaded SXB to /home/exabeam or anywhere on the master node except /opt/exabeam_installer.

Start a new terminal session using the exabeam credentials (do not run as ROOT).

Initiate a screen session. This is mandatory and will prevent accidental termination of your session.

 screen -LS [yourname]_[todaysdate]

Change the permission of the file using:

chmod +x Exabeam_[product]_[build_version].sxb

Execute the following command:

./Exabeam_[product]_[build_version].sxb fresh_install

/opt/exabeam_installer/init/exabeam-multinode-deployment.sh

The following are prompts based on the product you are installing.

1. The minimum number of nodes you need must match your product selection and sizing. The example below is for a large multi-node environment where Node 1 (master), Node 2 (worker) and Node 3 (worker) are designated for management functions. Node 4 onwards are worker nodes that process data. A single-node environment will be the same process, except you will only be asked for the IP address of one node.

   Warning

   If you are deploying a cluster with 21 nodes or more, three of the nodes will be automatically assigned as the role of host management node. The first three nodes will be designated host management nodes.

   Which product(s) do you wish to add? ['uba', 'ml', 'dl', 'cm'] dl
   How many nodes do you wish to add? (minimum: 2) 3
   What is the IP address of node 1 (localhost/127.0.0.1 not allowed)? [node1_address]
   What are the roles of node 1? ['lms_master', 'lms_slave'] : lms_master
   What is the IP address of node 2 (Localhost/127.0.0.1 not allowed)? [node2_address]
   What is the IP address of node 3 (localhost/127.0.0.1 not allowed)? [node3_address]
   What is the IP address of node 4 (localhost/127.0.0.1 not allowed)? [node4_address
   

   This IP assign step will repeat until all nodes are assigned addresses.

Valid credentials (SSH Keys) are needed for inter-node communications. The example below uses an internal path for the path to the SSH Private Key, which you must replace with your own. The path to the SSH Private Key must be an absolute path. Follow instructions:

1. Follow these instructions if you already have an SSH Private Key. This is the preferred method. Contact your Exabeam representative if you need assistance.

   The nodes within the Exabeam cluster communicate with each other regarding the processing status of the jobs, health status of the services etc. Valid credentials (ssh keys) are needed for secure inter-node communications.
   
   Do you have a ssh private key that can be used for internode communications? (If you don't have one, answer 'n' and we will create one for you. If you are running Exabeam on Amazon Web Services, you need to use the SSH key that the instance was launched with.) [y/n] y
   
   What's the path to the ssh private key? [/opt/exabeam_installer/.ssh/key.pem] /home/exabeam/.ssh/key.pem
   
   What's the user name used to deploy the public ssh key? This user must exist and have sudo power. [exabeam] exabeam
   
   Does Exabeam need password or SSH key to log in to all hosts? (This credential is needed only to put the SSH key on the machines. All communications moving forward will use the SSH key.)
   1) password
   2) SSH key
   ['1','2']: default (none): 2
   
   What's the path to the ssh private key? [/opt/exabeam_installer/.ssh/key.pem] /opt/exabeam_installer/.ssh/key.pem

2. Follow these instructions if you need to generate an SSH Private Key. This method is not supported for AWS and GCP deployments.

   The nodes within the Exabeam cluster communicate with each other regarding the processing status of the jobs, health status of the services etc. Valid credentials (ssh keys) are needed for secure inter-node communications.
   
   Do you have a ssh private key that can be used for internode communications? (If you don't have one, answer 'n' and we will create one for you. If you are running Exabeam on Amazon Web Services, you need to use the SSH key that the instance was launched with.) [y/n] n
   
   We will generate a new ssh key for the deployment at /opt/exabeam_installer/.ssh/key.pem
   
   What's the user name used to deploy the publish ssh key? This user must exist and have sudo power. [exabeam] exabeam
   
   Does Exabeam need password or SSH key to log in to all hosts? (This credential is needed only to put the SSH key on the machines. All communications moving forward will use the SSH key.)
   1) password
   2) SSH key
   ['1','2']: default (None): 1
   You will be prompted several times for password.
   Password: [password]

The installation will automatically partition your drives. However, if auto-detection fails, you will be prompted to manually configure your partitions.

You will be given a suggested storage layout, which you can accept or override.

Unable to autodetect drive types for host
Check if drive configuration/override is needed.

You will be given a suggested storage layout, which you can accept or override. If you choose to accept the auto-suggested drive mapping, type y and then proceed to the next step. If you choose to map the drives yourself, type n and follow the prompts to configure your drives to match the parameters in the table below.

To manually configure your drives, apply the parameters for the role and node you have assigned your host:

EX[appliance_type] mapping applied.

{
    [suggested drive mappings]
   ...
    [suggested drive mappings]
}
Please review the above, would you like to apply this drive mapping  automatically to the host? (Use lsblk or fdisk to verify on a separate screen) [y/n] n

To map an EX3000:

Please specify the drive purpose.
We typically put SSDs on the LVM for services requiring fast I/O (data, mongo, es_hot), and HDDs for dedicated services like hadoop, elasticsearch, kafka.
Ideally your host should have a mix of SSDs (fast) and HDDs (slow), so you should set your drive purpose accordingly to the Exabeam appliance specs.
Important: If your host has all SSDs mounted, please mark the drive purpose for dedicated mounts, and the rest for the LVM.
The size of the drive should be a good indicator as to which purpose it should be assigned to (larger sizes go to the dedicated mounts).
Important: you should not provision all your disks to the LVM, or the dedicated mounts, there should be a mix.


{'device': '/dev/xvdb',
 'driver': 'xvd',
 'model': 'Xen Virtual Block Device',
 'size': '1031GB',
 'table': 'unknown'}

1) Provision device /dev/xvdb to LVM (for data, mongo, or es_hot)
2) Provision device /dev/xvdb to dedicated mounts (for hadoop, kafka, or elasticsearch)
 ['1', '2']: default (None): 1
{'device': '/dev/xvdc',
 'driver': 'xvd',
 'model': 'Xen Virtual Block Device',
 'size': '1031GB',
 'table': 'unknown'}

1) Provision device /dev/xvdc to LVM (for data, mongo, or es_hot)
2) Provision device /dev/xvdc to dedicated mounts (for hadoop, kafka, or elasticsearch)
 ['1', '2']: default (None): 1
{'device': '/dev/xvdd',
 'driver': 'xvd',
 'model': 'Xen Virtual Block Device',
 'size': '2147GB',
 'table': 'unknown'}

1) Provision device /dev/xvdd to LVM (for data, mongo, or es_hot)
2) Provision device /dev/xvdd to dedicated mounts (for hadoop, kafka, or elasticsearch)
 ['1', '2']: default (None): 2

Select Option 2 for the remainder drives.

The following values are recommended.

1. For Advanced Analytics :

   How many elasticsearch instances per host? [2]  1
   What's the replication factor for elasticsearch? 0 means no replication. [0] 0
   How much memory in GB for each elasticsearch for each instance? [16] 16
   How much memory in GB for each kafka instance? [5] 5

2. For Data Lake:

   Note

   If you are choosing an instance type where the memory is greater than 120 GB, we require 4 warm nodes. Otherwise, you will receive a warning message during the deployment process.

   How many elasticsearch instances per host?  [4]  4
   How much memory in GB for each elasticsearch master node? [5] 5
   How much memory in GB for each elasticsearch hot node? [16] 16
   How much memory in GB for each elasticsearch warm node? [22] 22
   How much memory in GB for each kafka instance? [5] 5
   

   The following values are recommended for AWS and GCP deployments.

   How many elasticsearch instances per host?  [4]  4
   How much memory in GB for each elasticsearch master node? [5] 5
   How much memory in GB for each elasticsearch hot node? [16] 16
   How much memory in GB for each elasticsearch warm node? [22] 11
   How much memory in GB for each kafka instance? [5] 5

NTP is important for keeping the clocks in sync. If you have a local NTP server please input that information. If you do not have a local NTP server, but have internet access, use the default pool.ntp.org. Only choose none if there is no local NTP server and no internet access.

What's the NTP server to synchronize time with? Type 'none' if you don't have an NTP server and don't want to sync time with the default NTP server group from ntp.org. [pool.ntp.org] pool.ntp.org

The installation will automatically detect and assign a default route for your cluster.

Let us determine the right network interface name for the deployment.Discovered network interface name: eno1. This will be used as the default nic in the cluster.

If you would like to add internal DNS servers, select y and add them here. If not, select n. Name resolution here impacts only Docker containers.

Would you like to add any DNS servers? [y/n] n

If there are any conflicting networks in the user's domain, override the Docker BIP and Calico subnets. Answer y if you want to override (example is given below) and n if you do not.

Would you like to override the docker_bip IP/CIDR (172.17.0.1/16)? [y/n] y
Enter the new docker_bip IP/CIDR (minimum size /25, recommended size /16): [docker_bip_ip/CIDR]
Would you like to override the calico_network_subnet IP/CIDR (10.50.48.0/20)? [y/n] n

For Data Lake installations with clusters of 21 or more nodes, the installation will automatically designate the first three nodes as host management nodes (including master). The remainder of the nodes are marked as worker nodes that process data. Network throughput between management nodes must be capable of no less than 10 Gbps. For example, in a cluster of 24 nodes, nodes 1, 2, and 3, will be designated as host management nodes (including master). The remainder 21 nodes will be worker nodes that will process data.

If you do not have enough nodes to deploy the host management feature, the following message will appear:

Not enough nodes to deploy management nodes.

If the RPM (YUM) packages delivered with your installation have aged 3 months, you will be prompted to update your packages. You can also choose the option to Install pre-approved CentOS package updates from the main menu. ONLY UPDATE USING RPM (YUM) packages provided by Exabeam inside your SXB package.

While this update process will exit the original fresh_install script, once you have run the YUM updates and your hosts have been rebooted, you can return to and complete the deployment process by logging into your master host and apply:

/opt/exabeam_installer/init/exabeam-multinode-deployment.sh

Then select the "Deploy Cluster" menu option.

Download Exabeam_[product]_[build_version].sxb version file from Exabeam Community that you want to upgrade to. Place it on the master node in a temporary directory. Do not place the SXB file in the /opt/exabeam_installer directory.

Run the command below to start a new screen session:

screen -LS [yourname]_[todaysdate]

Change the permission of the SXB file.

chmod +x Exabeam_[product]_[build_version].sxb

Start a new terminal session using your exabeam credentials (do not run as ROOT).

Initiate a screen session. This is mandatory and will prevent accidental termination of your session.

screen -LS [yourname]_[todaysdate]

Execute the following command:

./Exabeam_[product]_[build_version].sxb upgrade 

The system will auto-detect your existing version. If it cannot, then you will be asked to enter the existing version that you are upgrading from.

If your previous software had non-standard customizations, the following output displays to indicate that a backup of the corresponding configuration and/or service files have been created in the given location. You can refer to these backup files to restore the customizations to your upgraded software.