JWST will provide revolutionary capabilities to characterize small exoplanets and open up pathways toward the observational studies of exoplanet habitability. Particularly, the temperate planets with radii between 1.7 and 3.5 times the radius of Earth (i.e., temperate sub-Neptunes) will be among the planets first observed for atmospheric signatures. Some of the temperate sub-Neptunes may support liquid-water oceans if they do not have massive H atmospheres and are thus not too hot at the bottom of the atmospheres. The mass of the atmospheres is not directly measurable by transits but critical for habitability. We have developed a novel method to combine transmission spectroscopy and planetary chemistry modeling to determine if temperate sub-Neptunes are capped by thick H2 atmospheres or thinner atmospheres that could foster a habitable environment. The key revelation is that the condition to form a liquid-water ocean and that to achieve the thermochemical equilibrium are mutually exclusive. The dominant carbon and nitrogen gases are typically CH4 and NH3 due to thermochemical recycling in a massive atmosphere of a temperate planet, and those in a small atmosphere overlying a liquid-water ocean are most likely CO and N2, followed by CO and CH4 produced photochemically. NH3 is depleted in the small atmosphere by dissolution into the liquid-water ocean. These gases lead to distinctive features in the planet’s transmission spectrum, and a moderate number of transit observations with the James Webb Space Telescope (JWST) should tell apart a small atmosphere versus a massive one on planets like K2-18 b and LHS 1140 b. This framework indicates that temperate sub-Neptunes presents a near-term opportunity for in-depth studies of atmospheric chemistry and assessment of potential habitability on exoplanets.