Verified answer

I am assuming you mean new unpainted drywall, as on a previously painted wall, paint will not absorb into the drywall core itself. Even on new unsealed drywall, it will not absorb into the core. Either way, just let it air out a few days with windows open and until it is very dry to the touch. Mineral spirit is no more toxic than an oil based primer or paint, it just needs to be used in a well ventilated area until dry and the fumes go away. Then seal with a good water base primer and buy some new paint for the top coat.

Ouch. Yes, its toxic. Either move out and ventilate for months, or replace the drywall.