It is not known precisely when construction of Colchester Castle began, but it was probably started in the 1070s or 1080s on the orders of William the Conqueror. The Castle was sited on the great stone base of what had been the Roman Temple of Claudius. William was linking his authority to that of the Roman emperors before him.
The keep is the largest surviving example built by the Normans, measuring 46m by 33.5m. Its ground plan is shared in England only by the White Tower at the Tower of London which makes Gundulph, Bishop of Rochester the likely designer of both castles.
Faced with a lack of good quality building stone, the Normans used the ruins of Roman Colchester to provide most of the brick and stone that they needed. The Castle was built in two stages. The first phase keep was only one storey high as is shown by the battlement which can still be seen clearly on some areas of the outer walls. The second phase of construction from about 1106 was entrusted to Eudo, who served the first three Norman kings as Steward of Normandy. The walls were raised higher, though how high is still a matter for debate. The traditional view is that the Castle was a three storey building. However, new evidence and a re-appraisal of the arguments suggest that the Castle was always roughly at the height it is today.
There are important surviving Norman features still to be seen in the Castle. The entrance doorway, which was originally protected by a fore-building, has a fine arch which is partly built of Caen stone brought from Normandy. The Great Stair, rising in the south west corner tower, is the largest winding stone staircase in Britain. The first floor fireplaces are early examples of their kind and have Y-shaped chimneys that discharge through holes in the walls. The royal apartments were situated in the east gallery on the first floor, a fact marked by the presence of an adjoining chapel or crypt, and a toilet!
Surrounding the keep was a large bank and ditch, of which only the north and east sections survive substantially intact. Within this area, called the bailey, only the foundations of a chapel can still be seen, but there would have been many other buildings such as stables, storehouses and workshops. On the north side of the keep a lower bailey extended down to the town walls.
The Castle was a royal fortress throughout the Norman period. It only saw serious action once, in 1216, when King John had to retake it from a French occupying force which was supporting a revolt by the English barons.