Cloud ConnectorsCloud Connectors Troubleshooting

Log in to your Exabeam Cloud Connectors platform.

Select Settings > General > License.

Identify the version in the Version field.

Run the sudo systemctl cat sk4compose command to verify the working directory for the docker file. You can skip this and the following verification steps if you already know the memory requirements.

$ sudo systemctl cat sk4compose
# /etc/systemd/system/sk4compose.service
[Unit]After=docker.service
Requires=docker.service
Description=SkyFormation Compose
[Service]
WorkingDirectory=/opt/exabeam/data/sk4

In this example, the working directory is /opt/exabeam/data/sk4 which is common for cloud-delivered deployments. For on-premises deployments, the working directory is either /opt/sk4/ or /var/lib/docker/volume/. Swap any instances of <workingdir> for your path in the remaining steps.

To verify sk4appconnect ran out of memory, run the following command on the server that hosts the Exabeam Cloud Connectors platform.

$ sudo grep "OutOfMemoryError" <workingdir>/logs/appconnect*

This will display any out-of-memory errors in the sk4 logs. When sk4appconnect runs out of memory you may see a log similar to the following:

/opt/exabeam/data/sk4/logs/appconnect-appdatasync.log:java.lang.OutOfMemoryError: GC overhead limit exceeded
/opt/exabeam/data/sk4/logs/appconnect-appdatasync.log:INFO   [2021-03-27 23:36:01,180] [syncpersona-Sales Cloud-Salesforce Production-sync-persona-3696335] com.skyformation.appdatasync.dataretrieval.AccountTaskWorker:call:366 - Task complete [TaskToExecute{id=3696335, executor=007f084d-2905-4bf5-8016-41e681fde1ff, definitionName=sync-persona, params=GeneralParams{com.skyformation.task.domainobjects.v1.GeneralParams@7fd959b4, data=null}, timeout=PT1H, signalStopPagingAfter=PT45M}]; artifact [GeneralArtifact{taskId=3696335, definitionName='sync-persona', msg='OutOfMemoryError: GC overhead limit exceeded', status=500, createdAt=2021-03-27T23:35:55.799Z, runningTime=PT11M38.316S, count=0, earliestRetrieved=null, latestRetrieved=null}] ; working thread [syncpersona-Sales Cloud-Salesforce Production-sync-persona-3696335] 
/opt/exabeam/data/sk4/logs/appconnect-etcd.log:java.lang.OutOfMemoryError: Java heap space
/opt/exabeam/data/sk4/logs/appconnect-etcd.log:Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
/opt/exabeam/data/sk4/logs/appconnect-etcd.log:Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
/opt/exabeam/data/sk4/logs/appconnect-service.log:Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Run the sudo docker stats --no-stream command to verify how much memory that the sk4appconnect container is taking.

$ sudo docker stats --no-stream
CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
f9a731bcd52f        sk4appconnect        0.10%               5.046GiB / 14.53GiB   34.73%              18.2GB / 13.6GB     0B / 0B             184
99428f0efc78        sk4siemconsumer      0.26%               1.604GiB / 14.53GiB   11.04%              2.82GB / 5.23GB     0B / 0B             86
[...]

In this example, the above output shows sk4appconnect currently takes 5Gib of memory.

Run the free -h command to verify the available memory.

$ free -h
              total        used        free      shared  buff/cache   available
Mem:            29G        8.3G        8.5G        1.5G         12G         19G
Swap:            0B          0B          0B

Make a copy of docker-compose.

$ sudo cp <workingdir>/docker-compose.yml <workingdir>/docker-compose.yml.backup

To add more memory for sk4appconnect, edit the Djdk.http.auth.tunneling.disabledSchemes definition in the docker-compose.yml file.

For example to add 1 GB of memory, the value would change from -Xmx3G -Xms3G to -Xmx4G -Xms4G.

# SK4_VERSION=2.5.160 by default SAAS CC sk4appconnect was allocated with 3G memory
/opt/exabeam/data/sk4/docker-compose.yml
container_name: sk4appconnect
    depends_on:
      - kafka
      - sk4etcd
      - nats
    environment:
      - >-
        JAVA_OPTS=-Dsk4TestMode=false
        -Djavax.net.ssl.trustStore=/data/application-connect/conf/sk4cacerts
        ${PROXY_OPTIONS} -Djava.net.useSystemProxies=true
        -Djdk.http.auth.tunneling.disabledSchemes= -Xmx4G -Xms4G
        -Xloggc:/logs/gc-appconnect.log -verbose:gc -XX:+PrintGCDetails
        -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation
        -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=10M
        -XX:CompressedClassSpaceSize=128M -XX:MaxMetaspaceSize=512M
        -XX:MetaspaceSize=512M -XX:+HeapDumpOnOutOfMemoryError
        -XX:HeapDumpPath=/logs/appconnect-mem-dump.hprof
      - SKYFORMATION_ENC_KEY=KtseFMbelGQInT63g1FVqIOeYD6IGaj4
      - SK4_VERSION=2.5.160

Save your changes when finished.

Run the sudo systemctl restart sk4compose command to restart sk4compose and load the new docker-compose.yml file.

$ sudo systemctl restart sk4compose

SSH into the machine which hosts the Exabeam Cloud Connectors platform.

Run the sudo -i command to run the shell with root-level credentials.

Run the cd /opt/exabeam/data/sk4/conf/shiro command to change to the shiro directory.

Run the cp client-shiro.ini client-shiro.ini.bak.beforehsts command to make a backup of the client-shiro.ini file.

Use you preferred text editor to make the following edits to the client-shiro.ini file:

Below the [main] section, add a new section:

# configure Shiro's default 'ssl' filter to enabled HSTS:
ssl.enabled = true
ssl.port = 8443
ssl.hsts.enabled = true
ssl.hsts.includeSubDomains = true

Below the [urls] section, replace the existing configuration and set /login.html = ssl, sk4Authc.

/index.html = ssl, sk4Authc, roles[sk4-admin]
/onboard.html = ssl, sk4Authc, roles[sk4-admin]
/logout = ssl, logout
/ = ssl, sk4Authc, roles[sk4-admin]
/** = ssl, anon

Save and exit the file.

Run the sudo systemctl restart sk4compose command to restart sk4compose.