It depends on how bad it is. Is it just touching? Then maybe not a problem. On the other hand, people have damaged flues bashing them with firewood. I hear my pans bang and groan as I fill it or empty it, so the weight of the sap will change things in there. I'm guessing you've got a couple thousand dollars put into those pans vs. a few dozen dollars put into the board and bricks. It's also wide open and easy to work on right now. Personally, I'd take the bricks out. They are only needed in the firebox.

The mortar is not as strong as you think. It shouldn't do much damage to your board. Maybe I'm not picturing what you mean. Do you have mortar between the board and the bricks?