OP, I should add one more thing. While I don't have experience with multiple Volvo transactions, I do with other auto mfgrs across more than a dozen special orders, and suspect the situation is the same: Given a choice, not all sales people and dealers like to handle special orders, and both would rather complete a sale sooner than later whenever they can.
Special orders take more time and elongated customer interaction. I suspect some number of buyers change their mind over the months-long waiting process, want a refund for reasons beyond the dealership's control, and effectively waste their time. When a customer bails and the special order is already in-flight, the dealer may then feel stuck with an odd configuration that may not sell as well or fast to someone else when it eventually comes in and sits on their lot. Special orders also require an increased level of precision a sales person or dealership may just not prefer, or are expert-enough, to deal with. (Don't ask about one of my special-ordered cars, where I didn't have visibility to a more rare factory-only installed package that was inadvertently left out by the GM on their internal ordering system --and not double-checked by the newer sales person-- until nearly 6 months later on the morning I arrived for pick-up, when I found the discrepancy as I was sitting in the finance manager's office looking closely at the paperwork before signing on the dotted-lines, and had to start-over reordering another MBZ.)
Some sales people (and perhaps dealerships) care more about moving product and having cash in-hand, vs. establishing a long term relationship where the customer gets exactly what they want to be most satisfied and will hopefully return to buy another car one day, let alone use their dealership for ongoing service. FWIW, I may have struck gold with my present Volvo dealership that appears to be more of the latter, having been owned by the same family for IIRC 28+ years, with my sales person and to-be service advisor each having been employed there for more than 27.
I know it was very clear to my sales person from the first few minutes of our interaction that I have a history of buying only what I want, not what I can more immediately get -- even if I have to wait and go through more hoops; Pressure techniques will not get a sale from me, and I must have a deal that just feels right -- not one where I have to go into hard negotiation mode or will feel like I've been taken advantage of later on. In your case, with the dealer playing hard ball on special order price, it may be because they don't want to do your special order; they are not sure about your commitment to really take delivery several months from now; they think you really just want "a XC60" and may be more flexible to buy color/features they have to sell now; or they believe they've geographically gotcha with a hot model they can sell all they can obtain from the factory.
As has been said, if price is most important, timing of your purchase may make a difference, but even more so, your willingness to compromise on what you want and buy off-the-lot will likely cost you less. OTOH, if you're more passionate like me and care about each color, feature and option to be exactly what fits your desire, you're going to almost always pay more for that choice. How much that may be is anyone's guess, and why I almost never talk dollar figures or percentages with anyone else. It is a complex set of choices only you can make in the end. Out here, we can easily throw out anonymous opinions, but we're not in your exact same situation. Again, good luck with your decision.