Brown bears (Ursus arctos) are threatened by habitat loss, habitat fragmentation by infrastructure and human settlements, and have been hunted to local extinction in large areas of their former range. We analyzed the habitat use during the non denning period of 106 radio-collared bears in an expanding bear population in Sweden in relation to resorts and towns, terrain ruggedness, sex and age of bears. Bear use increased substantially with increasing distance to towns and resorts for comparable habitat and terrain ENGINEs, also for independent scat surveys using DNA-analyses. More than 74% of all female bear locations were in the 29% of the terrain classified as “rugged” and located >10 km from any town or resort, whereas similar habitat closer to towns or resorts was avoided. Bears closer to larger settlements and resorts (<10 km) were on average 27–51% younger than in areas beyond (mean 4.4 ± 0.4 versus 8.9 ± 0.8 years for males and 4.4 ± 0.4 versus 6.0 ± 0.2 years for females). Sub-adult bears (<4 years) comprised up to 52% of all bear use within 10 km from resorts and settlements, likely representing exploratory dispersing individuals. These areas, however, contained only 8% of the old males (>7 years), the remaining 92% located beyond 10 km from major resorts and settlements. Recreational resorts are developing rapidly, typically near national parks, and may thus limit expansion or fragment existing bear habitats. Together with active conservation, safeguarding undeveloped corridors of forest and rugged terrain may be important for successful recolonization of the brown bear into its original range.