Open Broadcast Studio is a program to allow live mixing of video and audio sources. One might use it to direct a digital broadcast of a play using several live camera feeds and sending the result to Youtube.
The Encoder setting here selects the mechanism that will produce the final compressed video. All four settings, if working, will produce x264 compressed video.
The default software encoding will work on most hardware.
The low-cpu software encoding will work on less-capable hardware, at the expense of generating larger output sizes (for the same quality).
The hardware (QSV) encoding requires your CPU to be one of a few kinds of Intel CPUs with support for this feature.
The hardware (NVENC) encoding requires you to have one of a few kinds of NVidia graphics cards with support for this feature.
There is a comparison here:
which suggests that the default software encoding is the best, followed by NVENC, followed by QSV, with the low-cpu usage software encoding being the worst quality.