A Word Of Caution…
RFID Antenna designs can require optimsation for the particular RFID tags you are using, especially if you are using a small antenna coil. When evaluating a RFID solution, especially a components based solution, don't just assume it will all work. Test using the tags you plan to actually use, test the maximum and minimum read range (sometimes non optimal antennas can stop working if the tag gets too close) and test with the different mounting conditions you expect. I.e. if your reader might get mounted to a metal surface, near metal objects, near noisy electronics, etc, then make sure you simulate this in your testing to be sure you are not in for any nasty surprises. RF can be hard!
Antenna Coils
Wire wound antenna coils do not need to be round. Arranging them in a rectangular shape is fine as long as the corners are rounded off (90 degree bends are not good).
Best range
Best range is often achieved using a coil based antenna (PCB's antennas don't perform as well). However the size of a coil antenna should be selected based on the power of a RFID reader as simply going up in coil size reaches a point where the antenna can become too big for the power provided and is no longer optimal (you can get a 'donut' effect in the emitted RF).
Mounting to metal surfaces
Not good for RFID! Adding distance between the antenna and the metalwork will help.
Ferrite Backing
This can help improve antenna performance with surrounding metalwork. The ferrite backing is often a simple 1mm thick sheet which can be cut and mounted either close behind the antenna (cut to the same size or very slightly bigger), or with a bit of seperation (e.g. a couple of cm's) and cut to be bigger than the antenna (will often give better performance).