The small bone was the admission that Brozza was human

(see the damned winky face thing)
You have the write up in retrospect of what delayed DAD, there were a few more times Brozza delayed production 3 or 4, minor stuff really. But when you're trying to make a mountain out of mole hill, you'll scream he was getting too old when he delayed shooting until the prune juice he requested was brought to his trailer.
Brozza's pending hip replacement was not part of the reason they canned him. If aches and pain were a reason to fire someone then Craig would already be gone. He goes on about aches and pains more than Angela Lansbury talking about Bufferin.

(see the damned winky face thing)
Enough humor for ya?
When Brozza said he doesn't know why they fired him, he doesn't. At 57 he is still in great shape. They were talking about one more movie in 2004 to 2005. Which Eon had promised him and the public, all but in writing. Which turned out was essential in making them live up to their word.
In retrospect Craig at 38 needed more makeup than a drag-queen to go out to public event where photographers might get a candid shot. And Brozza was still doing ads for high end apparel and accessories
What happened in downtime was Dana Broccoli dies and immediately there after Babs goes nuts. Before 2004 she spent a year developing a Jinx spin off before MGM killed it. Brozza was talking to directors and writers with ideas for Bond pushing the next movie forward. Purvis and Wade, the Amos 'n' Andy of writing, claim they had a take on CR before Brozza and Tarantino (2003) came Babs and Mikey with one. Curious since in 2003 they thought it was impossible to film.
A lot of thing happened, saying Brozza just got too old isn't anywhere near the crux of the matter.