I have trained a custom object detection model using 750 images using ImageAI on Google Colab Pro about a month ago using TensorFlowGPU 1.13 and have roughly 30min/epoch training time. Now, when I train using the same dataset but with TensorFlowGPU 2.4.3 (ImageAI doesnt support old TF anymore), I am getting very little GPU usage (0.1GB) and 6 hour per epoch training times. I have tried training the same model on my local machine and I am getting very slow training times as well.
I am using the following imports (based on ImageAI documentation):
!pip install tensorflow-gpu==2.4.0 keras==2.4.3 numpy==1.19.3 pillow==7.0.0 scipy==1.4.1 h5py==2.10.0 matplotlib==3.3.2 opencv-python keras-resnet==0.2.0
!pip install imageai --upgrade
I am pulling my training data from Google Drive.
Is there anything I could be missing that could speed up my object detection training times on either Google Colab or my local machine? The slow training times is slowing my research down.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…