Drop the sword = free action Spell = action Take the sword back = object interaction Voila. You casted a spell with a sword in hand.

Some DMs could be a bit tight and say you need object interaction to touch your focus, but me as a DM a free hand is enough.