VirnetX Holding Corp. persuaded a U.S. appeals court to toss invalidity decisions on two of its patents, giving the company a chance for a potential billion-dollar payout from Apple Inc. over secure communications.

The U.S. Court of Appeals for the Federal Circuit vacated decisions from the Patent Trial and Appeal Board that the patents were invalid and remanded the cases for further proceedings.

A federal jury in Texas had found that Apple’s VPN on Demand, a virtual private network service, infringed the two patents, and the Federal Circuit earlier had upheld that decision. Questions on whether the patents should have...