William Tenn: THE MEN IN THE WALLS: Humans living like vermin

There are giant, technologically superior aliens who have conquered Earth. People live like vermin in holes in the insulation material of the walls of the homes the monsters have built, sneaking out to steal food and other items from the aliens. A complex social and religious order has evolved, with women preserving knowledge and working as healers, while men serve as warriors and thieves. For the aliens, men and women are just a nuisance, neither civilized nor intelligent, and are generally regarded as vermin to be exterminated. This story begins 'Mankind consisted of 128 people. The sheer population pressure of so vast a horde had long ago filled over a dozen burrows.'