When initially entering the game, there will be the familiar server list of all the possible servers you can join. These are categorized on the left. There are 3 public server categories: Official, Community, and Modded. Click on the Modded tab to show a list of available modded servers that you can join. Likely, the names of the modded server will have info on what mods they use.
No, you do not need to install any special programs or such in order to play on modded servers. The regular Rust application already has everything you need to run a modded game.(Keep in mind these are the same people who made Gmod, so modding is nothing new to the devteam)