Rangers, who are still searching for their first away win of the season, came from behind to take the lead only to see Scot McGleish score from the spot to see two points slip away.
Paul Furlong netted twice for the R's before Colchester levelled proceedings but Holloway believes that a poor start made the visitors' job all the tougher.
"I felt we just didn't get started. We tried to organise them but we let the first 25 minutes just go past. We played well for three-quarters of the game. But the start was our undoing."
Holloway admitted that a half time rollicking saw the R's come out a different side.
"I wasn't happy and I was as nasty as I get. We've got to keep clean sheets. We score enough goals but we keep letting them in."
The manager was however able to take the positives from the game and especially Arthur Gnohere's debut in the centre of defence.
"There were an awful lot of pluses. Arthur looks like he'll be a hell of a signing. He has a lot of quality.
"This is a group of players that are learning and we're getting better. We scored two fantastic goals.
"There is a long way to go in the season and after 46 games i'll talk to you all about where we are in the league."
Rangers slipped a place to fourth but entertain Wycombe Wanderers at Loftus Road this coming Saturday (3pm).