If you took the rocker cover off you would see that it has 4 holes in it where the spark plugs go through it to screw into the head as you already know. Around each of these holes there is an oil seal between the head and the underside of the rocker cover.
Also around the outside edge of the rocker cover there is a rubber seal/gasket. So 5 seals/gaskets in total under the rocker cover. It will be one of the seals/gaksets around the offending plugs.