If your seat belts are heavily soiled you may need to use a water based cleaning product in order to effectively lift all of the dirt out. Spray either a designated upholstery cleaning product, or a diluted all purpose cleaner onto a damp micro fibre towel and begin to work into the seat belt. Try to avoid applying the product directly onto the seat belt as this is when they can become overly wet.
For stubborn areas a stiff bristled detailing or nail brush can be used to work the product in and help lift the dirt out. Use it gently and ensure you don't overwork the area or brush the edges of the belt. Once cleaned, a vacuum cleaner can be used in conjunction with a crevice tool attachment or similar to extract any excess moisture and product residue from out of the seat belt.
To aid the drying of seat belts after cleaning you should pull them out and fix them into their anchor points which exposes them to the air instead of being coiled up in the mechanism whilst still damp.