Neither the original publication: Henry Cavendish, "XXI. Experiments to determine the density of the earth." Philosophical Transactions of the Royal Society of London, Vol. 88, 1798, pp. 469-526 (scan online), nor the annotated German translation: Henry Cavendish,"Versuche, um die Dichtigkeit der Erde zu bestimmen." Annalen der Physik, Volume 2, Part 1, 1799, pp. 1-62 (scan online) mention a separate building; they both just mention a room, presumably in an existing building.
As I was convinced of the necessity of guarding against this source of error, I resolved to place the apparatus in a room which should remain constantly shut, and to observe the motion of the arm from without, by means of a telescope;
From 1782 to 1810 Cavendish resided at a mansion: Cavendish House, Clapham Common, a picture of which is here. This was in the south of London, in the Borough of Lambeth. Best I can establish, this is where Cavendish conducted his experiments. For example, the Encyclopedia Britannica states:
Following his father’s death, Henry bought another house in town and also a house in Clapham Common, to the south of London. The London house contained the bulk of his library, while he kept most of his instruments at Clapham Common, where he carried out most of his experiments.
In this picture of the house shortly before, or during, its demolition it appears to have been constructed from brick masonry.
The book: Christa Jungnickel and Russell McCormmach, "Cavendish — The Experimental Life, 2nd ed.", Berlin: Max-Planck-Gesellschaft zur Förderung der Wissenschaften 2016 contains the following description:
Cavendish’s house has been called a mansion, but a better description of it from the time is “a tolerable good house, built with red brick.”
In comments OP raises the possibility of the experiment having been conducted in an outbuilding. The only specific reference to an outbuilding at Cavendish House that I could find in Jungnickel / McCormmach during a quick perusal is to a greenhouse mentioned in Cavendish's notes on experiments on air. Modern greenhouses consist largely of glass, which would rule them out as the location of the experiment, but methods of construction used for greenhouses may have been very different during Cavendish's time.
An 1877 map appears to show a group of three outbuildings at the far end of the Cavendish House property, but whether any of these existed in 1798 is impossible to know.