I would not necessarily limit this to instruments only. Here are a few of my thoughts:
1) Yes, some instruments play in a "cash register" this is close to the human voice and therefore, might create problems underneath dialogue. I'd try to not use too many instruments which have their best register in the octave starting from middle C as this is the range where the human voice is very present. However, this doesn't mean that you cannot write for these instruments underneath dialogue. The clarinet, for example, works perfectly in that register but it's kind of close to the frequencies of the human voice. And therefore, here's number 2.
2) Movement! If you want to occupy a similar frequency space between the human voice and an instrument, regard the instrument as a counterpart voice to the protagonist. Write in between the sentences and create a little overlap between those two voices when they call for attention from the listener. Or even have only little movement at all. The less your melody moves, the less it will compete with dialogue.
3) Melody complexity. Simple lines like scale tone movement tend to work better in this context. Generally, I refer to writing small adjacent intervals. In those moments where you have pauses in the human voices, go for bigger leaps and let the instrument take over the attention for a second.
4) Register. The easiest thing is to stay out of the register of the human voice and control your dynamics. Write your lines below that octave starting from middle C or above it. Either way, make sure that you don't step into the extreme registers like VERY low or VERY high as these also call for attention. Often times, these registers are used for effects like growls, runs, squeezes, etc. ... this might be distracting.
5) Changes in color. I'd be careful with bringing in different instruments and changing the color as this will be noticed by the audience. As long as the story doesn't call for a change in color, don't change it because it will confuse the listener. However, on the other hand, this is very powerful to create "musical paragraphs" to help tell the story. This concept also works beautifully with occupying different registers.
I hope that helps! Go for it! :D