For Blootooth to work the devices have to be paired, which normaly involves a challenge/responce sequence of a 4 digit pin number on the receiving device (to prevent random others connecting to your equipment).
Also, the broadcast name function can be turned off, preventing others from knowing your device even has bluetooth enabled, and thus preventing them from connecting unless they've previously slept with your wife/girlfriend/mistress to find out what the bluetooth device name is.
Some functions, like file/photo sharing and push messaging, do not require pairing but you can't use those protocols to control someone elses device.
Edit to add: Another method, used by PS3 controllers, involves connecting the devices together with USB first at which point the pairing is done automatically with no user input required.