I have the same issue and found the following in the Boss Knowledge Base (item 201978699):
USB Functionality:When using the BR-800 as an audio interface or control surface, all incoming audio connected to inputs 1 through 4 is mixed together and is seen as a single stereo audio signal in recording software--the BR-800 will be seen as "L/BR-800" and "R/BR-800" in software.
Note: There's not a function to pass more than a stereo mix from the BR-800 to a computer in real-time. The ability to record 4 independent tracks simultaneously is on the BR-800 itself. However, after recording 4 tracks simultaneously on the installed SD memory, these 4 tracks can be transferred onto recording software...