There is no evidence that BYRON met the Ladies, yet he obviously knew of them by reputation — as this letter shows.
In a letter to Elizabeth Bridget Pigot, July 5, 1807:
“Since my last letter I have determined to reside another year at Granta, as my rooms, etc., etc., are finished in great style, several old friends come up again, and many new acquaintances made; consequently my inclination leads me forward, and I shall return to college in October if still alive. [….] Edleston and I have separated for the present, and my mind is a chaos of hope and sorrow. To-morrow I set out for London: you will address your answer to ‘Gordon’s Hotel, Albemarle Street,’ where I sojourn during my visit to the metropolis… I rejoice to hear you are interested in my protege; he has been my almost constant associate since October, 1805, when I entered Trinity College. His voice first attracted my attention, his countenance fixed it, and his manners attached me to him for ever. He departs for a mercantile house in town in October, and we shall probably not meet till the expiration of my minority, when I shall leave to his decision either entering as a partner through my interest, or residing with me altogether. Of course he would in his present frame of mind prefer the latter, but he may alter his opinion previous to that period;–however, he shall have his choice. I certainly love him more than any human being, and neither time nor distance have had the least effect on my (in general) changeable disposition. In short, we shall, put Lady E. Butler and Miss Ponsonby to the blush, Pylades and Orestes out of countenance, and want nothing but a catastrophe like Nisus and Euryalus, to give Jonathan and David the ‘go by.’ He certainly is perhaps more attached to me than even I am in return. During the whole of my residence at Cambridge we met every day, summer and winter, without passing one tiresome moment, and separated each time with increasing reluctance. I hope you will one day see us together. He is the only being I esteem, though I like many…”
Published in The Works Of Lord Byron, Letters and Journals, vol. 1, ed. by Rowland E. Prothero (1898) (available at Project Gutenberg), the editor put in the following footnote regarding Sarah and Eleanor:
“Lady Eleanor Butler (c. 1745-1829), sister of the seventeenth Earl of Ormonde, and Sarah Ponsonby (circ. 1755-1831), cousin of the Earl of Bessborough, were the two ‘Ladies of the Vale,’ or ‘Ladies of Llangollen.’ About the year 1779 they settled in a cottage at Plasnewydd, in the Vale of Llangollen, where they lived, with their maidservant, Mary Caryll, for upwards of half a century. They are buried, with their servant, in the churchyard of Plasnewydd, under a triangular pyramid. Though they had withdrawn from the world, they watched its proceedings with the keenest interest.”
The footnote also included a partial letter written by Mrs. Piozzi:
“If,” writes Mrs. Piozzi, from Brynbella, July 9, 1796, “Mr. Bunbury’s ‘Little Gray Man’ is printed, do send it hither; the ladies at Llangollen are dying for it. They like those old Scandinavian tales and the imitations of them exceedingly; and tell me about the prince and princess of ‘this’ loyal country, one province of which alone had disgraced itself” (‘Life and Writings of Mrs. Piozzi’, vol. ii. p. 234).