Yhyjyk

The Rebel fleet at the end of Rogue One (even after sustaining many losses) still seems considerably larger than the fleet in A New Hope.

General Dodonna essentially answers this question during the briefing:

The battle station is heavily shielded and carries a firepower greater
than half the star fleet. It's defenses are designed around a direct
large-scale assault. A small one-man fighter should be able to
penetrate the outer defense.

Knowing that the Death Star is on the way to the Rebel base, and knowing that a direct assault would be a useless waste of resources, the battle plan sticks exclusively to only those resources that have a chance of making any difference. Also - all of the capital ships didn't necessarily make it to Yavin 4:

"The remaining Alliance forces on Yavin 4 were a ragtag
collection....almost no capital ships had survived the battle at
Scarif"

- Sparks: From a Certain Point of View

Though it does appear that some ships did escape Yavin 4 prior to the battle:

"[I] Don't think they're planning to evacuate," Quersey said. "Most of
the equipment left with the fleet. It's all in the hands of the pilots
upstairs."

- Duty Roster: From a Certain Point of View

We don't see the Rebel Fleet in A New Hope, there are no Captial ship or Escorts that feature in the film.

Apart from the Tantive IV that is captured at the start of the film of course.

We only see the forces posted on Yavin, it's possible that a portion of the remaining Rebel forces fighters are with the larger ships and only some on Yavin.

There is a large time gap between episode 4 and Rogue One. During this time, it is entirely possible that other battles have weakened the rebel fleet.