git - GitLab runner gets "host key verification failed" for submodules

Question

I am facing the following issue:
I am trying to configure a GitLab CI pipline (shell).
My repository contains two submodules.
Both submodules are on the same GitLab server as the super repository that contains them.
The clone is an SSH clone, I have configured the keys locally, and also added my key to GitLab.
On the machine where the runner is installed, I can clone everything with no issues including the submodules.
However when the runner is trying to clone, it returns with "host key verification failed" but only for the submodules.
I have tried configuring the runner both with its own user and with my user :

sudo gitlab-runner install --user=<user> 

to no effect.
What confuses me the most is that the error is only for the submodles even though they are on the same server as the super repo that contains them, and the super repo can be cloned with no issue (when I turn off the submoudle recursive var in the yml file):

GIT_SUBMODULE_STRATEGY: recursive

But then of course I don't have the submodules.
I will be grateful for any suggestions on what to check or try!

Answer 1

"host key verification failed" is about the machine key (the ones listed in ~/.ssh/known_hosts), not your gitlab key.

For example : if the initial git clone is not run with the same user as the one which updates the submodules, then they do not have the same ~/.ssh/known_hosts file, and the ssh command could work for the first user without error, while failing for the second.

---

The clean fix would be : copy the host key you know to be correct to the expected known_host file.

see for example ssh use known_hosts other than $HOME/.ssh/known_hosts :

# use a 'known_hosts' file with the host key of your git server
GIT_SSH_COMMAND=`ssh -o UserKnownHostsFile=<some provisioned known_hosts file>`

The workaround almost everybody uses is : turn HostKey verification off.

# for example : set the GIT_SSH_COMMAND environment variable
GIT_SSH_COMMAND='ssh -o StrictHostKeyChecking=no'