The Constitutional Court annulled the selection of Conrado Reyes, who has been in the post for less than a month. The court's decision comes days after a judge heading a UN-backed commission investigating corruption in Guatemala accused Mr Reyes of having links with criminal gangs.
Mr Reyes has denied the allegations. Judiciary crisis
The allegations against Mr Reyes were highlighted on Monday by Spanish judge Carlos Castresana, who heads the International Commission against Impunity (CICIG).
Mr Castresana said the Guatemalan government had not kept its promise to reform the justice system, and announced he was resigning as a result. He accused Mr Reyes of having links with "illicit organizations" and urged President Alvaro Colom to dismiss him.
President Colom said that in a meeting on Thursday, Mr Castresana had presented him "firm evidence" supporting the allegations against Mr Reyes. The president did not give details of the alleged evidence, but said it showed that people around Mr Reyes had links to "parallel organizations".
President Colom, who had appointed Mr Reyes, said he was satisfied with the court's decision. In its ruling, the court did not address directly the accusations leveled against Mr Reyes.
The head of the constitutional court said the judges had made their decision in order "to overcome the institutional crisis" which had engulfed the judiciary. The Guatemalan parliament will now have to convene a commission to select six new candidates from which President Colom will choose a new attorney general.
The International Commission against Impunity says 98% of crimes committed every year in Guatemala go unpunished because the justice system is too weak or too corrupt.