I too don't use this model (Duo-Capture EX in my case) but I suggest you first of all follow the Roland instructions for setting the input level on the Duo Capture itself (probably tweaking up the level control until the peak light just flickers then back off a tad). Then try the follow rear-panel settings -
The Input Monitor Off: in this case you will need to select Input Echo in the Sonar track to hear anything. The disadvantage is that the sound will be fractionally delayed (but in most recent computers and using ASIO drivers this will not be too distracting). The advantage is that you will be monitoring exactly what Sonar will record.
Hi-Z On: if you plug your guitar directly into the jack socket in the Duo Capture, otherwise it should be Off.
Input Gain set Lo: play the guitar and verify that the track level meter registers a healthy dose of green bars. If not, try it on Hi. Proceed with whichever setting gives you lots of green but avoid constant amber or any red.
(Note that if you change Hi-Z or Input Gain you may have to tweak the input level control again. You may also need to make slight adjustments here to achieve a healthy signal on the Sonar track meter. Just make sure that you are not seeing the peak light at all on the Duo Capture.)
Then make a trial recording. At playback, keep an eye on the playback meter in the Sonar track. The meter should pretty much show the same levels as you saw at record, providing you have the track level (Volume) control set to 0. (You will only see the playback meter reading if you disarm the record button on the track during playback). Hopefully, you will now hear a pretty much perfect record of what you played.
If the playback still does not sound like what you played (or if the input monitoring set up above does not allow you to hear what you expect), then it may be worth checking the signal routing out of the PC to your ears, as I don't think you mentioned how you are monitoring playback.