Hey!
I'm unsure about your issue specifically, but I looked through your pastebin post and noticed a couple things that might be worth checking.
It seems that you're spawning a new thread to run a blocking PortAudio stream. Perhaps instead you could create a non-blocking stream and let portaudio kick off the new thread for you? Bencina, the author of PortAudio, recommends using the non-blocking stream where possible - he goes into a bit more detail on why here. You can find a rust-portaudio non-blocking stream example here. I'd be interested to hear whether this change fixes your issue, as the blocking stream tends to be a little less stable (which I imagine is partly because it's used a lot less than the non-blocking stream).
Also, it seems like you're planning on sharing data with the audio thread using a Mutex (i.e. the Arc<Mutex<IpData>>). Audio threads tend to be the epitome of real-time, and can easily glitch or underflow if they get stuck waiting for access to locks or other system resources or lose thread priority due to waiting, etc. Bencina has a great post on this called Real-time audio waits for nothing. I find rust's std::sync::mpsc::channel to be a nice tool for non-blocking communication with the audio thread. (Edit: I doubt this is your problem btw, as both threads barely seem to touch the lock).
You might already be aware of all this, but just thought I'd mention them in case 