Does the engine itself seem to have poor idle quality to the point where it may stall?.... or is it possible that you are simply feeling too much of the engines vibrations due to worn motor mounts? Failed motor mounts (which is common due to age) can allow too much of the engines vibrations to be transferred to the body of the vehicle. This might be something to consider. The fact that you have a high idle when cold leads me to believe your IAC valve (valve that controls idle speed) is in good proper working condition.
Certified Toyota Technician