I do this sort of thing quite a lot.
Their might be several reasons for muddiness or pedals not workiong like you expect.
The audio path should be Sonar -> interface->pedal->interface. Sonar’s external effects plugin can be useful for this but not essential.
The interface->pedal connection should use a line out output on the interface. It might be that you are overloading the pedals, in which case you need to lower the volume of the output to them. Either by using the interface’s controls to do that or lowering the track volume in Sonar. Sometimes both are necessary.
Another problem might be an impedance mismatch. Some pedal circuits are designed so that they only work properly if they are connected to a guitar or something else with a similar impedance. Examples are pedals like the Fuzz Face and many other older fuzz designs, where the guitar itself becomes part of the pedal’s circuit.
The only way to get such pedals to work properly is to feed them what they are designed to work with, which means line level low impedance outputs such as those on interfaces won’t be able to drive the pedal properly.
Sometimes pedals and other external effects don’t work as you would expect because the audio going into them doesn’t have the harmonic compexity of a guitar. For example, running a synth into a distortion/fuzz pedal or chorus/phaser/flanger pedal has been done for decades but not all such pedals do a good job of it. Some add hardly any distortion or noticable effect at all, it’s a matter of trying different ones to see which work.
Personally I find the MXR DIstortion+ can work quite well, though it needs both knobs set pretty high and sounds smoother than it does with a guitar. The EH Deluxe Electric Mistress can work, and most delays are fine. Big Muffs sound very muddy and some phasers do so little they may as well not be there.