Sentence lengths have declined. The average sentence length was 49 for Chaucer (died 1400), 50 for Spenser (died 1599), 42 for Austen (died 1817), 20 for Dickens (died 1870), 21 for Emerson (died 1882), 14 for D.H. Lawrence (died 1930), and 18 for Steinbeck (died 1968). J.K Rowling averaged 12 words per sentence (wps) writing the Harry Potter books 25 years ago.
J. K. Rowling is the wrong comparison here, since the Harry Potter books were written first for children and then (in later volumes) for young adults. For recent writers, the proper comparisons to Austen, Dickens, and Emerson would be Don DeLillo, Toni Morrison, and Joan Didion.
The bigger problem is the failure to recognize the developments in the punctuation of English prose over the centuries. Chaucer and Spenser wrote before the current roster of punctuation marks — in addition to periods and commas, we now have colons, semicolons, parentheses, and dashes, which add often subtle variety of meaning, variety of clausal relations, without necessarily affecting comprehensibility — became clearly established. Ancient texts were typically written in scripta continua: not only did they lack punctuation, there were not even any breaks between words. In such a regime of writing, there’s nothing even to correspond to the word “sentence.”
(Indeed, an important subset of the history of written language is the history of punctuation: see e.g. Malcom Parkes’s wonderful, and wonderfully titled, Pause and Effect. Also brilliantly titled, and a fine read: John Lennard’s history of parenthesis in English verse, But I Digress.)
Also, consider this famous passage from Hemingway:
In the fall the war was always there, but we did not go to it any more. It was cold in the fall in Milan and the dark came very early. Then the electric lights would come on, and it was pleasant along the streets looking in the windows. There was much game hanging outside the shops, and the snow powdered in the fur of the foxes and the wind blew their tails. The deer hung stiff and heavy and empty, and small birds blew in the wind and the wind turned their feathers. It was a cold fall and the wind came down from the mountains.
One could re-punctuate it thus:
In the fall the war was always there, but we did not go to it any more. It was cold in the fall in Milan and the dark came very early; then the electric lights would come on, and it was pleasant along the streets looking in the windows: there was much game hanging outside the shops, and the snow powdered in the fur of the foxes and the wind blew their tails, the deer hung stiff and heavy and empty, and small birds blew in the wind and the wind turned their feathers — it was a cold fall and the wind came down from the mountains.
That would make the sentences longer, but it wouldn’t make them more sophisticated or challenging. It would just make them a bit worse.