How to populate object having array of object in mongoose?

I want to populate the adminId path to User Model.
Here is the code

adminInfo: {
    _id: false,
    adminId: [{
      type: Schema.Types.ObjectId,
      ref: 'User'
    }]
 }

I have tried using .populate('adminInfo.adminId') but it's giving empty array [] whereas .populate('adminInfo') giving array of admins ids but not getting populated to User model