Site CollectorSite Collector Administration Guide

Prepare the GCP VM for installation.

Complete the environment requirements and prerequisites. 

Create the VM.

To create the VM you will choose the appropriate GCP project, and begin the configuration of a new instance.

1. Log in to the Google Cloud Platform (GCP) console with your registered credentials.

2. On the top left corner, from the Select a project list, select the GCP project for which you want to create your new VM for Site Collector installation.

   

   In this example, the selected project is NGSC Experiments. 

3. Navigate to Compute Engine in the left pane and go to VM instances. Or navigate to VM instances by accessing https://console.cloud.google.com/compute/instances.

4. Click Create Instance. 

   

5. On the Create Instance page, enter the following information.

   

   • Name – Enter a name for your VM.

   • Region – Select the region and the associated zone.

   • Machine Configuration – Select the machine configuration details. For example, 4 vCPU, and 16 GB memory

Configure the operating system and disk setup.

In the Boot Disk section, click Change and select the following configuration options. 

Configure advanced configuration options for networking.

1. Click Show Advanced Configuration > Networking.

2. To open the ports for internal communication, add network tags in Advanced options. Make the network ports mentioned in the prerequisites accessible.

3. In the Network Interfaces section, select a shared network interface; for example, platform-shared-vpc-us-west1. If the list of shared subnetworks is not available as a selection, contact your support team.

   

Allocate additional storage for Site Collector.

In the Disks section, add the following disks required for Site Collector installation. Refer to Site Collector Specifications for minimum storage requirements.

The Disks section displays the disks that you added. 

Configure SSH keys for secure access to the VM.

1. Copy the SSH key from the SSH Keys section. 

   

2. Paste the SSH key in the Security section. 

   

On the Create Instance page, click Create.

Open an SSH session to connect to your VM instance. You will need your private key and the IP address of the instance.

Use the commands to install the required packages for your operating system version:

RedHat 7

# install packages
sudo yum update -y
sudo yum install -y jq rsync screen curl openssl gawk ntp ntpstat

# enable NTP
sudo systemctl start ntpd
sudo systemctl enable ntpd

# install docker
sudo yum install -y yum-utils
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install -y docker-ce docker-ce-cli containerd.io docker-compose
# NGSC v1.10+ will also require docker-compose-plugin
sudo yum install -y docker-compose-plugin


sudo systemctl start docker
sudo systemctl enable docker

# verify that the docker is installed correctly
sudo docker run hello-world

# install tmux
curl --silent https://storage.googleapis.com/ngsc_update/exa-cloud-prod/tmux_install.sh | sudo bash

RedHat 8 and 9

# install packages
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo dnf install 'dnf-command(upgrade)'

sudo yum update -y
sudo yum install -y jq rsync screen curl openssl gawk 
sudo yum install -y chrony

# install docker
sudo yum install -y yum-utils
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install -y docker-ce docker-ce-cli containerd.io
# NGSC v1.10+ will also require docker-compose-plugin
sudo yum install -y docker-compose-plugin

sudo systemctl start docker
sudo systemctl enable docker

# verify that the docker is installed correctly
sudo docker run hello-world

# install docker-compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/bin/docker-compose
sudo chmod +x /usr/bin/docker-compose
# verify that the docker-compose is installed correctly
docker-compose -v

# install tmux
curl --silent https://storage.googleapis.com/ngsc_update/exa-cloud-prod/tmux_install.sh | sudo bash

Install additional packages required to run Site Collector:

Use the following commands to install required packages such as docker, jq, rsync, screen, curl, and openssl (for the full list, see prerequisites). You must ensure the packages you install meet the minimum version requirements, if specified.

# install package
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg lsb-release jq rsync screen curl openssl gawk ntp ntpstat

# enable NTP
sudo systemctl start ntp
sudo systemctl enable ntp

# install docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose

sudo systemctl start docker
sudo systemctl enable docker

# for docker-compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo chmod 755 /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

# verify that the docker is installed correctly
sudo docker run hello-world

Check the disk names by running the following commands.

[exabeam@ngsc-oncar-1 ~]$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   80G  0 disk
├─sda1   8:1    0  200M  0 part /boot/efi
└─sda2   8:2    0 79.8G  0 part /
sdb      8:16   0  200G  0 disk
sdc      8:32   0   50G  0 disk
sdd      8:48   0   25G  0 disk

Create partitions by running the following commands.

sudo parted -s /dev/sdb mktable gpt
sudo parted -s /dev/sdc mktable gpt
sudo parted -s /dev/sdd mktable gpt
sudo parted -s /dev/sdb mkpart ext4 1MiB 100%
sudo parted -s /dev/sdc mkpart ext4 1MiB 100%
sudo parted -s /dev/sdd mkpart ext4 1MiB 100%

Create the file system by running the following commands.

sudo mkfs.ext4 /dev/sdb1
sudo mkfs.ext4 /dev/sdc1
sudo mkfs.ext4 /dev/sdd1

Create mount directories for NiFi and mount partitions according to the disk size specified in the prerequisites.

Use the following commands to create the folders and mount the disks to them:

sudo mkdir /content_repository
sudo mkdir /provenance_repository
sudo mkdir /flowfile_repository

sudo mount /dev/sdc1 /content_repository
sudo mount /dev/sdd1 /provenance_repository
sudo mount /dev/sdb1 /flowfile_repository

Following is the df output example.

[exabeam@ngsc-oncar-kanitkar-1 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  8.5M  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sda2        80G  3.1G   77G   4% /
/dev/sda1       200M   12M  189M   6% /boot/efi
tmpfs           1.6G     0  1.6G   0% /run/user/1000
tmpfs           1.6G     0  1.6G   0% /run/user/0
/dev/sdb1       197G   61M  187G   1% /content_repository
/dev/sdc1        50G   53M   47G   1% /provenance_repository
/dev/sdd1        25G   45M   24G   1% /flowfile_repository

Set up auto mount for your drivers.

Check mounted drivers' IDs with the ls -lha /dev/disk/by-uuid command as follows.

[exabeam@ngsc-oncar-1 ~]$ ls -lha /dev/disk/by-uuid
total 0
drwxr-xr-x. 2 root root 140 Sep  8 19:07 .
drwxr-xr-x. 8 root root 160 Sep  8 18:41 ..
lrwxrwxrwx. 1 root root  10 Sep  8 18:42 0308-0EB7 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Sep  8 19:07 2ef7c5ee-4e95-4c5d-b254-70874ac5626e -> ../../sdc1
lrwxrwxrwx. 1 root root  10 Sep  8 18:42 34e26dc7-0242-4153-9621-9e70eaac573d -> ../../sda2
lrwxrwxrwx. 1 root root  10 Sep  8 19:07 7fb561d7-0781-492d-aeec-d3e0f6f84b35 -> ../../sdd1
lrwxrwxrwx. 1 root root  10 Sep  8 19:07 bb625665-ddfd-423f-8878-ec91169b95ae -> ../../sdb1

Edit your /etc/fstab file to add the three drivers that are linked to the repository folders. For example:

Restart the server.

Verify that the VM instances is ready for Site Collector installation.

Run the following Site Collector pre-check script to validate the system. 

curl -Ss https://storage.googleapis.com/ngsc_update/exa-cloud-dev/pre-check.sh \
     | bash -s

The following message indicates that the instance is not ready and pre-checks failed.

The following result indicates that the pre-checks and configuration are successful and the VM is ready for Site Collector installation.