GUEN - 2022-03-23

I have setup vm host as a controller and build both slurm and munge. Installation seems OK with no error and daemons started OK (slurmctld, slurmdbd). However when I do sinfo from host controller itself it says can not communicate:

[root@localhost slurm]# sinfo
slurm_load_partitions: Unable to contact slurm controller (connect failure)

I initially set hostname in the slurm.conf but caused dns error so I directly put IP adderss and got errors above:
I am wondering why it can not?
I am wondering if this has anything to do with the way daemon is running. If I get daemon status periodical it seems to be exiting and entering the running over and over, is it normal?? (below):

Mar 22 21:49:05 localhost.localdomain systemd[1]: Started Slurm.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:06 EDT; 12ms ago
 Main PID: 27827 ((lurmctld))
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27827 (lurmctld)

Mar 22 21:49:06 localhost.localdomain systemd[1]: slurmctld.service: Service RestartSec=1s expired, scheduling restart.
Mar 22 21:49:06 localhost.localdomain systemd[1]: slurmctld.service: Scheduled restart job, restart counter is at 1709.
Mar 22 21:49:06 localhost.localdomain systemd[1]: Stopped Slurm.
Mar 22 21:49:06 localhost.localdomain systemd[1]: Started Slurm.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:07 EDT; 3ms ago
 Main PID: 27851 ((lurmctld))
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27851 (lurmctld)

Mar 22 21:49:07 localhost.localdomain systemd[1]: Started Slurm.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:08 EDT; 3ms ago
 Main PID: 27868 ((lurmctld))
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27868 (lurmctld)

Mar 22 21:49:08 localhost.localdomain systemd[1]: Started Slurm.
Mar 22 21:49:08 localhost.localdomain systemd[1]: slurmctld.service: Main process exited, code=exited, status=203/EXEC
Mar 22 21:49:08 localhost.localdomain systemd[1]: slurmctld.service: Failed with result 'exit-code'.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:09 EDT; 1ms ago
 Main PID: 27885 ((lurmctld))
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27885 (lurmctld)

Mar 22 21:49:09 localhost.localdomain systemd[1]: Started Slurm.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:10 EDT; 1ms ago
 Main PID: 27902 ((lurmctld))
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27902 (lurmctld)

Mar 22 21:49:10 localhost.localdomain systemd[1]: Started Slurm.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:11 EDT; 1ms ago
 Main PID: 27919 (systemd)
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27919 /usr/lib/systemd/systemd --switched-root --system --deserialize 18

Mar 22 21:49:11 localhost.localdomain systemd[1]: slurmctld.service: Service RestartSec=1s expired, scheduling restart.
Mar 22 21:49:11 localhost.localdomain systemd[1]: slurmctld.service: Scheduled restart job, restart counter is at 1714.
Mar 22 21:49:11 localhost.localdomain systemd[1]: Stopped Slurm.
Mar 22 21:49:11 localhost.localdomain systemd[1]: Started Slurm.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:12 EDT; 4ms ago
 Main PID: 27929 ((lurmctld))
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27929 (lurmctld)

Mar 22 21:49:12 localhost.localdomain systemd[1]: Started Slurm.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:13 EDT; 2ms ago
 Main PID: 27942 ((lurmctld))
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27942 (lurmctld)

Mar 22 21:49:13 localhost.localdomain systemd[1]: Started Slurm.
 slurmctld.service - Slurm
   Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-03-22 21:49:14 EDT; 3ms ago
 Main PID: 27958 ((lurmctld))
    Tasks: 0 (limit: 100918)
   Memory: 0B
   CGroup: /system.slice/slurmctld.service
           └─27958 (lurmctld)

Mar 22 21:49:14 localhost.localdomain systemd[1]: Star