We consider the geometry of quantum states associated with random matrix Hamiltonians belonging to ensembles that exhibit an integrable-to-chaotic transition in terms of the nearest-neighbour energy level spacing distribution, focusing, specifically, on the β-Gaussian ensembles with generic Dyson index. In the case that the total Hamiltonian contains a single parameter, the distance between two states is captured by the fidelity susceptibility, whereas, when the total Hamiltonian contains multiple parameters, this distance is measured in terms of the quantum metric tensor. For the tridiagonal Gaussian β-ensemble, which shows chaos-to-integrability transition with varying Dyson index, we first calculate an analytical expression for the ensemble-averaged fidelity susceptibility for a two-by-two matrix representation of the Hamiltonian for generic values of the Dyson index, and show that it diverges as the total Hamiltonian, which is the sum of a diagonal matrix with independent elements and a tridiagonal β-matrix, goes over to the integrable phase. Next, for large-dimensional matrices, we numerically compute the fidelity susceptibility and the quantum metric tensor components, respectively, for a one-parameter and a two-parameter class of Hamiltonians that we construct from the β-ensemble, and find scaling relations of these quantities for chaotic and integrable phases. We also consider other variations of the β-ensemble, such as the one that preserves the rotational invariance of the ensemble, and compute the relevant ensemble-averaged fidelity susceptibility to show that it has similar features as the tridiagonal ensemble in the integrable as well as the chaotic phase, thereby establishing the universality of these properties. Finally, the presence of non-vanishing non-diagonal elements, which arises due to the rotational non-invariance of the β-ensembles, is a specific feature of the corresponding metric tensor, and we use this to illuminate the difference between the quantum state space geometry of these ensembles and the rotationally invariant ones, such as the classical Gaussian ensembles.