A personal choice, guided by local tradition.
I once heard someone say that they have a white button up shirt that they only wear in the temple. Therefore they distinguish between "street clothes" and "temple clothes." I, for one, don't consider my white shirt as everyday wardrobe, so I don't adhere to that thought.
On the other hand, when I was an ordinance worker in Provo, we were advised not to wear our white suits and shoes into/out of the temple for our commute.
The only guidance I have ever read is on the color and design of temple whites. How you accomplish that is up to you. If I worked in a kitchen where my uniform was white pants and white shirt, I probably wouldn't double-task that set of clothes for the temple. But baptismal ordinances and temple ordinances seem an appropriate extension for a set of white trousers.