Who Really Wrote the Works of the British Renaissance?

Hi All: I want to draw your attention to a new interview we've just posted on the LibraryThing blog, with scholar Anna Faktorovich, in which we discuss her new project, the British Renaissance Re-Attribution and Modernization Series.

ER participants will recognize this name, as the author's main study, as well as some of the newly available texts from the period that she has presented in the series, were offered as giveaways during this past month's ER batch.

Dr. Faktorovich discusses her new computational-linguistic model, and how she has used it in the study of 284 works of the British Renaissance, coming to the conclusion that these works were written by the same six ghostwriters.

Come read her argument, and tell us what you think!

https://blog.librarything.com/2021/12/an-interview-with-scholar-anna-faktorovich...

For example, the data indicates only two linguistic signatures between the three Bronte sisters, suggesting it is likely the initial assignment of these texts to only two male brothers was more accurate than the current belief three women wrote them.

Hmmm. I think that's the point where you would normally discard the method as unreliable and try something else.

It's "interesting", but I'm inclined to to agree with >2 thorold: that this is dubious methodology.

Amazing what you can 'prove' if you play around with numbers long enough.

What a load of bollocks.

>1 AbigailAdams26: I think LibraryThing's spotlighting the argument without providing counter arguments is embarrassing and a bit threatening, considering the risks to us if responses boost the interview.

>7 aspirit:

This author also spammed the forums, which is another reason not to highlight her work. Such behavior should not be rewarded.

"the “Shakespeare” plays and poetry translated into Modern English registered as a separate linguistic signature from these same texts in their original spelling" = "My versions of Shakespeare read like they were written by me". The idea that the style of a translation can be used to identify the original author is so bizarre it barely qualifies as an idea.

"I learned about the £2,400 loan William and his brother Henry Percy (Earl of Northumberland) took out from Arthur Medleycote (London merchant tailor) in 1593, just before the granting of the theater duopoly by Elizabeth I in 1594. This documented proof, without any corresponding record of what else William could have spent this sum on, firmly establishes that William re-invested this sum in troupe-development and theater-building in London" - no, it establishes nothing of the sort. It only establishes that you don't know what he did with the money. Given Percy was imprisoned for debt more than once, and we know that his brother paid for his food in prison, a more likely use would be to satisfy his latest creditors.

"As I mentioned, none of William Percy’s plays or poetry, or the plays I am re-attributing to him in this part of the series, have ever been translated into accessible Modern English before" - Rowse years ago did his "Contemporary Shakespeare" which while not entirely modernising the language does make it eminently accessible.

I think the claim for Percy as Shakespeare is little more than the usual snobbish refusal to accept that a bright middle-class boy from an ambitious family who went to a decent provincial school could go on to make a success of himself, and so to claim an aristocratic candidate instead. It's an attitude that's been around for years and seems to me to shew a remarkable ignorance about Elizabethan society.

As for the Brontës being two brothers, we really are ascending to Nephelokokkygia.

I really don't think LibraryThing should be promoting this.

This brings to mind how the science fiction magazine Astounding went down after the editor stated to push Dianetics and psionics.

>1 AbigailAdams26:
This is the literary equivalent of those linguistic conspiracy theorists who derive all languages from Hebrew, or who claim that Serbian / Sanskrit / Tamil are the oldest language and that all others derive from them. There's a few nationalistic Hungarian foot soldiers out there connecting the language of Linear A to Hungarian.

Faktorovich's arguments run along similar lines: the establishment stinks; she's uncovered a massive cover-up or at least some sort of centuries-long blind spot; her ideas have extremely wide-ranging implications while vindicating a downtrodden minority; she ought to be recognized as a revolutionary in the field; Great Scott, the lone maverick's done it again!

At several points during this interview she says that for anyone to even criticize her arguments, they'd have to read, work through and refute several other of her books. ("You really have to read Volumes 1-2 to understand how overwhelming the evidence is for these conclusions, as reading this summary alone could not possibly convince anybody that the history with which they are familiar is entirely incorrect"). Right. In reality, her methodology is faulty and the whole thing is built on the bane of much computationally-supported research: GIGO.

Digital humanities is a fairly young field, and there's no doubt that applying Big Data and computational methods like stylometry will allow us to answer questions we can't without those methods. Callaway's fun experiment in untangling the two authors of Good Omens is a good example of that. Faktorovich's work, in contrast, looks, walks and talks like quackery.

ETA: I don't think this is the sort of thing that LT should be highlighting. Yikes!

I find it disappointing that if she is reading an average of at least 6 books a day, she can't even name one or two of them.

Wow, this is all kinds of wrong. Why is LT promoting this person's dodgy work?

Just at the common sense level --- can you really believe that all the works of the English renaissance were written by just six people?????

Where is proximity1 when we need him?

>12 MarthaJeanne: Yeah, that question went way off course! I had at some point expected to look up her analysis of genre literature; but seeing where she goes with other topics, it’s probably quite worthless.

I agree with everyone else. It's one thing to interview an author with questions like "what challenges do you have when self-publishing scholarly non-fiction?" and another thing entirely to promote self-published claims as if they have inherent scientific merit without any kind of corroboration or peer review.

>15 abbottthomas: :D

>2 thorold: Here is an excerpt from my forthcoming article on this topic that explains this point in full:

The contracted phrase "I can’t" only appears in “Anne Bronte’s” Agnes and in “Emily Bronte’s” Wuthering, confirming that the signature behind Wuthering is at least partially “Anne’s”… "I did not" appears among the top-6 phrases in every one of the 5 tested “Bronte” texts (in 4 of these as the 1st or 2nd most frequent)—“Emily”, “Anne” and “Charlotte’s” —and only appears in one other text, oddly enough, in “Thoreau’s” Walden… Out of these anomalies, the least surprising should be the “Brontes” because their three eventually feminine bylines commenced under male pseudonyms of “Currer”, “Ellis” and “Acton Bell”. Only one of the sisters, Charlotte, had a basic education, and she is claimed to have taught her sisters how to write highbrow literature at home. The death of two of the Bronte sisters—Emily in 1848 and Anne in 1849—coincided with the re-attribution of the masculine “Bell” pseudonyms to the three feminine “Brontes”. While most modern critics claim the “Brontes’” fame was instantaneous and pre-dated the re-gendering, this re-marketing under feminine names has clearly been the main element that canonized the books attributed to them. My findings from the Renaissance indicate that borrowing the names of the recently deceased consistently succeeds in elevating the texts with these bylines to much greater heights in popularity than rival bylines of the healthy and living. Examples of this trend include “Christopher Marlowe” and “Philip Sidney”, both of whom only began having books released with their byline after their deaths. The “Brontes’” novel-publishing history begins in 1847 with the first printing of “Currer Bell’s” Jane Eyre by Smith, Elder. The claimed instantaneous popularity of Jane, inspired Thomas Coutley Newby to accept and to release a couple of months later the multi-volume set of “Ellis Bell’s” Wuthering and “Acton Bell’s” Agnes Grey; both were then re-printed under the “Emily” and “Anne” bylines and with “Charlotte” as the attributed editor with Newby’s press in 1850. Back in 1847, Newby’s first advertisements of Wuthering and Agnes claimed they were new novels by the same author as the author of Jane Eyre, while contradictorily specifying that Wuthering was by “Acton”, and not by “Ellis” (as he claimed on the 1847 title-page) or by “Currer” (as would have been consistent with the Jane’s-byline claim). And when Newby sold American copyrights, the first American edition of Wuthering carried the “Currer Bell” byline, again contradicting the 1847 British edition’s “Ellis” byline (Newman, B. (New York: Broadview Editions, 2007). Wuthering Heights, 33-4.). Arguably the first “Bronte” sister to receive a byline in this saga was “Charlotte Bronte” when her byline appeared just above “Currer Bell’s” in its second printing of Jane in December of 1847, but within the book her name only appears in association with the “Preface”, while even the dedication to Thackeray is from merely “The Author”. The 1849 edition of Shirley still carries the “Currer Bell, Author of Jane Eyre” byline; and the Leipzig, Smith and other 1850 editions of Jane still have “Currer Bell” in their bylines. The first mention of “Charlotte Bronte” as the author of Jane appears to be a strange bound set of two letters called “A memento of her friendship with Mrs. Elizabeth Gaskell her Biographer” dated 1851-3, and currently stored as a manuscript in the University of Leeds library. The numerous changes in these bylines have been dismissed as the outcome of women being censored from accessing publishers under their own names, and thus going to extraordinary length to confuse even their male pseudonyms to avoid being discovered as the underlying authors. But this gender-disguising subterfuge hypothesis is contradicted by the occasional public acceptance of the female bylines by the Bronte family, in parallel with the continuing mis-attributions to the wrong masculine “Bell” bylines of some of the still un-feminized texts. The publishing history of the “Bronte” novels is one of the most provable cases of ghostwriting, pseudonym-usage and mis-attribution that has not been seriously questioned by scholars because the simple re-gendering solution to this mystery fits with the propaganda of feminist progress for British women even in the nineteenth century, when they were still barred from enrolling into universities, from voting, and most other basic human rights./ With this publishing history in mind, we can now tackle the computational-linguistics attributions of the “Bronte” texts. “Unmasking” claims that “Charlotte Bronte’s” Professor matched both of her sisters. This assertion is false according to my data because The Professor is a strong 11-test match to “Charlotte’s” second tested novel, Jane Eyre, and it is a relative non-match at 6-9 tests to the other three tested “Bronte” texts. In my data, the main anomaly among the “Brontes” is not “Charlotte”, but instead “Emily’s” Wuthering Heights. Just as Newby had initially claimed that Wuthering was by the same author as Jane, there is an extremely strong similarity-match in 18-tests between these two novels, confirming that they share a single dominant author. Since the only other text that has been attributed to “Ellis Bell” (re-assigned to “Emily”) was the 1846 collection of poems that includes the tri-byline “Currer, Ellis, and Acton”; it logically follows that there are only two linguistic styles or two co-writers despite the triplicates of “Bronte” and “Bell” pseudonyms assigned to their texts. On my 28-tests, there is a clear set of matches between “Anne’s” Agnes and Tenant on 11-12 tests, and “Charlotte’s” Jane matches Professor on 11 tests. Wuthering appears to have been written by the linguistic-signatures of both “Anne” and “Charlotte” because it matches not only “Charlotte’s” Jane on 16-18 tests, but also “Anne’s” Tenant on 13 tests. It is possible that Charlotte and Anne Bronte were truly authors, but it is more likely that there were two male collaborating professional ghostwriters desperate to make a living, who were experimenting with multiple bylines until they struck fame with the “Brontes’” feminine names.

>4 MarthaJeanne: Unlike all previous re-attribution studies of the British Renaissance, I provide all of the raw data (https://github.com/faktorovich/Attribution) and explain my exact attribution method not only in the book, but also in this interview. In the article about the "Brontes", I explain how previous computational-linguistic articles, such as Moshe Koppel, Jonathan Schler and Elisheva Bonchek-Dokow’s “Measuring Differentiability: Unmasking”, have indeed manipulated the numbers and demonstrate the statistical errors of omission, misreporting, and false-data. In contrast, my method has so many layers of different tests that it is near-impossible to manipulate or bias the resulting data. I would be delighted to test any mystery test you propose to prove the method works. You just can't cling to the original bylines being the correct answer, and instead be willing to consider that a ghostwriter, a pseudonym or the like might have been involved.

>6 Crypto-Willobie: I do not understand this slang expression. Are you stating that I have an enormous load of testicles? Is this a negative because I am a woman and you are being ironic?

>7 aspirit: You are all making what I would imagine are intended to be counter-arguments here - unless "bollocks" is indeed intended as a compliment. The counter-arguments have also been made across the previous 400 years of scholarly misattribution of these texts. I cannot imagine LibraryThing would want to publish a hit-piece that insults folks as embarrassments and suggests they are threatened by scholarly research. I was hoping for some rational discussion in response to this interview, but I guess scholarly discourse has been replaced with insults.

>8 lilithcat: I did not spam the forum. I posted an invitation to discuss this topic in one place other than the Author Hobnob area, and my comment was immediately deleted by you, who also sent several messages to me explaining that you are glad you censored my ability to speak in public. If your deletion of my post was rational and it is no longer there, are you bringing it up to make it sound like I am doing something wrong to discredit me over nonsense when you have no rational basis or evidence to counter my arguments?

Even if the content of everything from that quarter to date didn't already support it, I'd look at the unbroken dump of >19 faktorovich: (and rapid-fire subsequent posts) with a growing suspicion of crankery. Still, I guess someone who suffers from this sort of logorrhea might "reasonably" conclude that vast swaths of literature were created by a much smaller number of authors than are reflected in their historical attributions.

While ordinarily I'd like to see the sort of intellectual charity exhibited by >1 AbigailAdams26:, I think AF's record on LT should forfeit it.

>9 DuncanHill: You do not understand my findings or what I said in the interview. First, other computational-linguistics scholars have claimed that the modernized "Lear" is by a different linguistic-author than the old-spelling "Leir". This point is not in contention. The statement you quote from me is saying that scholars have failed to recognize that erroneously comparing modernized with old-spelling versions has led to modernization being the reason these two and others suggest anonymous pre-"Shakespeare" versions of many of these plays were not by "Shakespeare". It is irrelevant if my versions of "Shakespeare" sound like me; in fact, I have made the same type of edits to all plays as those who have modernized "Shakespeare", so all of these Percy plays should now sound like Percy after modern editing, making "Hamlet" obviously recognizable as similar to Percy's self-attributed plays such as "Aphrodisia". I said the opposite of your claim that the style of a translation can be used to identify the original; in fact, I said that the original should never be compared for authorial-attribution with the modern version.

Here is a fragment from Volumes 1-2 where I explain the "food"-payment: The most common negative criticism levied by biographers against William Percy is that he had served time in debtor’s prison. In 1887, de Fonblanque appears to have become the first biographer to claim Percy spent time in Fleet Prison for debt without citing a supporting source. The debtor’s prison mention has been repeated in studies such as Gerald Brenan’s history of the Percys in 1902. Dodds imagines a scenario where William Percy would have been imprisoned after Henry was locked in the Tower, and creditors lost faith in William’s capacity for repayment without his brother’s backing. The piece of evidence that might have inspired de Fonblanque initial claim is the payment listed in the Accounts of the Earls of Northumberland in 1611/2 for £11 19s to “Mr. Bagwell the keeper of Oxford Castle for the charges of Mr. William Percy’s diet there.” Another £14.14s.4d. was charged for William’s “diet” in 1617-8. Dodds argues: “Oxford Castle was the town prison, and the most probable explanation of this entry is that William was imprisoned there for debt.” An alternative explanation for this expensive culinary bill is that William was working in an administrative capacity at this Castle, which also enclosed the administrative offices of this county. The Castle’s function changed to solely being a prison in the 18th century, before it was repurposed again into a museum in 1996. If William was imprisoned in this Castle, Henry’s bill would have included charges for the room, and other elements essential to make an extended stay bearable for a gentleman. Regardless of if this was an imprisonment or an employment, this payment confirms William resided in Oxford by 1611-2. Instead of considering William’s potential gainful employment at the Castle, biographers have been describing William’s time in Oxford as a retirement. Dodds traces this claim back to the largely fictitious or gossip-based account of Anthony Wood in Antiquities of the City of Oxford (1661-6?; 1889).

By "Rowse years ago did his 'Contemporary Shakespeare'" you seem to be referring to the Rowman series of semi-translations of "Shakespeare". What does this have to do with me creating the first ever translations of Percy's self-attributed plays or the anonymous plays. The only play this can apply to is the first quarto of "Hamlet" because the second quarto of "Hamlet" has been translated before, but as I explain in the book this first previously-untranslated "bad" quarto entirely changes the storyline.

It is absurd to make any re-attribution claim (for or against) based solely on the class of the author. My findings are that only six ghostwriters worked non-stop for extremely low payments across their lifetimes to publish this massive output of texts in the British Renaissance. Most of them did so because they were in desperate need for money, but Percy was acting to defend his family against the corruption that had executed/assassinated two previous Earls of Northumberland and also sent his brother (the new Earl) into the Tower. The problem with "Shakespeare" was not that he was poor, but rather that he was not a real person; "Shakespeare's" signatures are blatantly forged by Percy as they match the handwriting style in other linguistically Percy-authored texts.

LibraryThing exists to explain new relevant research with librarians. If all of their Renaissance shelves/books are misattributed, this is a cataloging problem that librarians have to be aware of. The rational approach to counter my arguments would be for you to actually read the full series first, as I am certain the overwhelming evidence in it would convince even the most staunch deniers.

>10 anglemark: If you have to post a public threat against an editor suggesting they will be fired for sharing a research-study you disagree with; you really must have no rational support for your absurd position. Scientology is a religion based on imagined fiction; my research study is based on overwhelming evidence including not only computational-linguistic data, but also forensic accounting, handwriting analysis and various other types of proof. Previous re-attributions of this corpus have indeed been psychic or intuitive, but my findings are the exact opposite of that as I make my case entirely based on a quantitative analysis and documentary evidence. I included the steps involved in my attribution method in the interview. You can test it on any texts of your own to see if you will confirm that it works or not.

>26 faktorovich: (re: >10 anglemark:)

That wasn't a threat, it was an historical observation.

>23 faktorovich:

>my comment was immediately deleted by you,

As I previously explained to you, I did not delete your comment. I have no power to do so. Your comment was flagged by multiple people, and, under the rules of the site, that resulted in its disappearance.

you censored my ability to speak in public

Once again, I have no power to do so.

Why you chose to violate the Terms of Service and then got upset when there were consequences to those violations, I cannot fathom. Had you just joined, perhaps other members would have cut you more slack. But you have been a member of this site for several years, and should have been familiar with, and respected, its norms.

>11 Petroglyph: The Hebrew language had a significant early impact on many world languages. And Sanskrit has been acknowledged to be an ancient language that also influenced those that followed it. What does any of this have to do with my research?

You seem to have concluded ironically that the establishment is made up of idiots who have had a blind-spot for the British Renaissance as they have been propagating the greatness of "Shakespeare" via his pro-monarchy plays. Yes, ghostwriters are the downtrodden minority, who have received little credit, while achieving many of the accomplishments their sponsors have been applauded for. They are like the Sherpas who are hired to guide the incapable westerners to the top of literary mountains, only to remain a nameless title themselves.

Those who were refuting Einstein's theory of special relativity had to at least claim to have read Einstein's paper(s) on this subject; so why would anybody have a right to ridicule and discredit any researcher without at-minimum reading the research they are refuting? You have not even mentioned the computational-linguistic attribution steps I included in the interview to explain what specifically is wrong with this precise approach.

There are several typos in the Callaway article you cite. My Volumes 1-2 include a chapter where I examined the structural story arcs of all of the "Shakespeare" plays and proved that the Percy/Jonson division can be seen in their plots as well as in their linguistics. My explanation is far more sophisticating and logical than what Callaway does and it addresses the rules of storytelling in the literature field, instead of using NSC classifications that are more typically applied to the statistical study of cell behavior in biology. You seem to be claiming that Callaway is superior simply because scholars have agreed he is superior, and anybody who questions his superiority is not worthy of consideration. Instead of saying nonsense about how I walk, look or talk (which you cannot see given our distance and the textual method of communication), why don't you actually point out what specific elements of my findings you find to be quacking?

Do you think the sort of post you have added here should be highlighted by LibraryThing instead? Maybe something that jumps between anti-Semitism, nonsense and insults against a woman's looks?

>12 MarthaJeanne: If you had read my response, you would see the link to my latest bunch of reviews I did for PLJ: https://anaphoraliterary.com/journals/plj/plj-excerpts/book-reviews-summer-2021. You can see the names of the books I am reading there. The problem is not that I cannot name them, but rather that you cannot be bothered to read their names.

>13 amanda4242: "All kinds of wrong"? Are you referring to the insulting nonsense in these responses? There is nothing "dodgy" about my work. The problem is in your lack of respect for fellow humans' scholarly labor.

>14 Crypto-Willobie: As I explain in the book, "Philip Henslowe's Diary" indicates that many plays were written in 2 weeks after commissioning. This would mean 26 plays per year, and 1,560 plays in 60 years that was the length of Percy's literary career. There were only around 400 plays published during these decades, so there is absolutely nothing unbelievable about my findings. What I find to be unbelievable is that anybody can believe a writer can write only a single play in a lifetime and then never publish another play again despite continuing to live across the following decades. It is also unbelievable that "Christopher Marlowe's" byline appeared for the first time a year after his death, and yet claims of his authorship of many texts are accepted as believable. So, read my series, and then you will find out if you believe in my six-writers conclusion or not. If you dismiss it without reading the evidence, and while continuing to believe the current nonsensical attributions... you are doing yourself a disservice.

>21 faktorovich: I question the competence of anyone claiming to write about the English language who does not understand the expression "what a load of bollocks".

>25 faktorovich: Not Rowman, Rowse. Any Shakespearean scholar will know him, even if they disagree with him.

>16 2wonderY: My research has been cited in 32 scholarly articles/books according to Google Scholar: https://scholar.google.com/citations?user=dJD72pMAAAAJ&hl=en. So, it is a fact that my analysis of literature has been appreciated by many, and it is untrue to call it "worthless".

I didn't know that "LibraryThing exists to explain new relevant research with librarians." as stated in >25 faktorovich:. Where can I find out more about this and does LT have other purposes that members may be unaware of.

>17 norabelle414: It is incredible how insulting your statement is without delivering any rational message. As I explain in my "Author-Publishers" book, the best British-American writers were also publishers/self-publishers, including Dickens, Woolf, Twain etc. etc. In the sciences, the best example is Galileo Galilei's "Dialogue Concerning the Two Chief World Systems" that he self-published before it was banned. Nobody in the establishment would have given a positive peer review to Galileo if he had pitched his Earth-revolves-around-the-Sun theory in contradiction with the previous idiotic model that the Earth was in the center of the universe. Galileo was found guilty by the Inquisition, so this is the company you are keeping if this is where you have chosen to go with your argument. The best science/literature has always been self-published because the establishment has always prioritized being celebrated for their brilliance over scientific facts, and the merits of true literary greatness.

>20 faktorovich: When these findings are explained by someone who can write clear English, is employed at a major university, and publishes in a major publishing house, then I might go into them. Until then, I have no intention of wasting my time. >17 norabelle414: Sounds very rational to me.

>24 paradoxosalpha: You are arguing that because I am a quick writer this means that nobody else can be quick at writing. This is obviously false and the opposite of the truth. The speed of my writing only proves the normal speed achieved by a professional writer/ ghostwriter. You are focusing on censoring my research without addressing what my research is.

>28 lilithcat: Who the other censors might have been is unknown, but that you censored me is a fact that you keep repeating. My post invited discussion about research and did not violate any Terms of Service, which cannot be proven or disproven now since you and the other censors deleted the post in question. So it is pointless to argue about what it was unless you still have it and want to repost it here.

>33 DuncanHill: Rowman is the publisher. I added a more specific citation for the series you were referring to, and explained why your reference to it was unrelated to what I was saying about a lack of prior Percy translations. You have now also misunderstood my clarification, while making it seem as if I have misunderstood something. I hope all who are confused by this will refer to my previous response to DuncanHill. I understand the expression "what a load of bollocks"; I used ironic confusion to show the absurdity of the underlying meaning of this insult.

>35 Taphophile13: What exactly do you think a website dedicated to the Library have to share if not research about library science, librarians, and other related subjects? Why would chatting about low-brow pop fiction be anywhere as significant as a scholarly discussion about the underlying authors behind the British Renaissance?

>34 faktorovich:

You may have been cited, but not always with approval.

>41 faktorovich: https://www.librarything.com/about

"LibraryThing is an online service to help people catalog their books easily."

Because, more people are interested in chatting about fiction than in the underlying...

>39 faktorovich:

that you censored me is a fact that you keep repeating.

No, it is a NON-fact that YOU keep repeating. I have said here, and in communications with you, that I have no power to censor anyone here, even if I wanted to.

I cannot understand how someone who considers herself a scholar can be so lacking in the ability to understand such clear statements.

>37 MarthaJeanne: You are making an appeal-to-authority logical-fallacy, together with a xenophobic insult directed at my Russian heritage without citing examples of any lack of clarity in my English. There is no rationally necessary order between a scholar publishing field-changing research and his/her employment by a "major university". In most cases, the ground-breaking research comes first, and the recognition of these findings and subsequent employment in the Ivy League comes afterwards. And two of my books have been published with McFarland (a major publishing house) nearly a decade ago. If you believe posting insults and slander is less of a waste of time than just reading the research in question; then, you obviously would not understand my books even if you read them.

>42 lilithcat: If you actually read the dissertations/books/articles that Google Scholar links to, you would find that all of these citations were with approval, including a special note of thanks to me for the review from an economist who said he adjusted the book before publication with help from my review of it in PLJ. I have never seen a negative comment about my work in a published text that mentions it.

>40 faktorovich: The publisher was University Press of America, before they took over Rowman and Littlefield. As far as I can tell it was not republished under the Rowman and Littlefield name - I think the last printings were in 1987, and the takeover in 1988. Using a publisher's name which does not appear on the works or in Cauveren's bibliography is needlessly confusing, and suggests an unfamiliarity with the work. Perhaps if you had taken a little more time you would have expressed yourself more clearly.

Perhaps you should refrain from saying you don't understand something if you don't want people to believe you.

>41 faktorovich: I mentioned nothing about pop fiction nor was I comparing the significance of anything to anything else. I am trying to find out where it says that the site is "dedicated to the Library" and that research, libraries and related subjects are the appropriate topics. Sorry if I don't meet those standards.

>43 MarthaJeanne: My study proves beyond-doubt that nearly all "author"-bylines across the British Renaissance are currently incorrect. So if finding this out is not something that eases or assists librarians with accurate cataloging of books; what would? And the number of comments in this thread so far clearly proves there is "interest" in discussing "the underlying..." or my study.

>44 lilithcat: I understand that you want to convince the group that I do not understand what you want to censor me, but have no power to do so, and you have not done so, but have also indeed done so.

>47 DuncanHill: As any researcher would have done, I looked up what you were referring to and came across this page: https://rowman.com/Action/SERIES/_/CSS/The-Contemporary-Shakespeare-Series. I mentioned Rowman simply to assist other readers with finding the series you were referring to, and not because it made any difference who the original publisher was. Why would you or I check Cauveren's bibliography to research the version of this series that is currently available for purchase/reading? As I explained the more important point is that this series is irrelevant to your argument that there have been translations of the texts in my series before.

>49 faktorovich: I haven't heard anyone here saying that they want to discuss your study. I have heard many voices saying that they don't think LT should have showcased it.

>50 faktorovich: If you again try to write about any of your books in Talk outside of Hobnob, several members will flag the post, and your post will disappear. This is what happens to all advertising on LT.

>48 Taphophile13: I was paraphrasing your own quote: "LibraryThing is an online service to help people catalog their books easily." You don't see the relationship between my paraphrase and this line?

>52 MarthaJeanne: Yes, most of the comments posted here are not discussing any of the facts I bring up in the interview or in my study. This is instead an attempt to censor LibraryThing from publishing the completed interview without any rational reason given for such censorship other than the demands of the insulting mob.

>34 faktorovich:

And how many of those entries in Google Scholar has been in the field of computational linguistic analysis?

>50 faktorovich:

The problem there is that everyone else posting here understands how LT works. How the flagging mechanism works. It seems that you do not.

>53 faktorovich: I did not say that; it was >43 MarthaJeanne: who said it and she was quoting from here: https://www.librarything.com/about. Please don't misattribute statements.

>55 andyl: As I explain in the interview, I have been working on related subjects to this series across the past 20 years. My "The Formulas of Popular Fiction" (11 citations) dissects both the structural and linguistic formulas of these texts, as do my other books like "Rebellion as Genre" (6 citations: dives into Scottish-English linguistics as well as the linguistics of the authorial styles), and "Gender Bias in Mystery and Romance Novel Publishing: Mimicking Masculinity and Femininity" (2 citations: explores the linguistic differences/similarities between male/female writing styles).

>19 faktorovich: The contracted phrase "I can’t" only appears in “Anne Bronte’s” Agnes and in “Emily Bronte’s” Wuthering, confirming that the signature behind Wuthering is at least partially “Anne’s”… "I did not" appears among the top-6 phrases in every one of the 5 tested “Bronte” texts (in 4 of these as the 1st or 2nd most frequent)—“Emily”, “Anne” and “Charlotte’s” —and only appears in one other text, oddly enough, in “Thoreau’s” Walden

Many of us on this site are authors or avid readers familiar with English-language publishing processes, historical and modern. We're really supposed to pretend we can't see what's wrong with these assumptions? (ETA: Or, worse, we were expected to argue against all of it?)

What was the LT team thinking with this spotlight?

(Please note, faktorovich, that I'm not asking for you to answer. I have no interest based on your behavior in these forums to converse with you.)

>57 Taphophile13: The wording of your message indicated that you were offended about me contradicting your earlier statement.

>60 faktorovich: You misread me. I am amused by all this and I think it is someone else who is taking offense.

The phrases "I can't" and "I did not" are referring to the top-6 most common 3-word phrases in these texts. The rest of the article explains this point. Authors tend to repeat some of their favorite phrases frequently. I prove this argument across Volumes 1-2 of the series as well as in this article about the "Brontes". My responses in this forum have been extremely polite and purely rational in comparison with the avalanche of insults hurled at me, and yet I would not "block" anybody for responding to my research even if such replies are nonsensical.

>61 Taphophile13: Given the wide range of insults posted, it would be very strange if I missed the intended offense entirely.

>19 faktorovich: It is possible that Charlotte and Anne Bronte were truly authors, but it is more likely that there were two male collaborating professional ghostwriters desperate to make a living, who were experimenting with multiple bylines until they struck fame with the “Brontes’” feminine names.

You seem to be missing the point we’re all making here, that extraordinary claims require extraordinary evidence. That is an extraordinary claim. It only becomes “more likely” if you have strong biographical evidence of the existence of the ghostwriters and their links to the Brontës. Otherwise, the accepted attribution, vouched for by a huge body of letters, memoirs, manuscripts and all the rest, has to stand, and at most you have exposed possible doubts about who wrote which text.

I’d guess that the conditions the siblings grew up in, playing writing games together and sharing most of their activities, would skew the data of their writing styles anyway.

Hi All: My apologies for dropping this notice last night and only returning to the discussion now. I wanted to get the interview out, and to bring it to members' attention, but in the intervening time have not been able to get onto LT. Saturday is usually my out and about day.

I've been reading through the thread, and I'm seeing two main things. First, that many LT members do not agree with the results of Dr. Faktorovich's study (to put it mildly), and do not think LT should have highlighted it, with our interview. Second, that the conversation between Dr. Faktorovich and other members has not been going well, with accusations of censorship and insult on one side, and of spamming and prolixity on the other.

I will try to go through and answer specific concerns raised, but on a general note, I want to say that the decision to present this interview is on me. Tim gave me approval, but I was the one who assented to it, and presented the idea to him. I did so because I thought the thesis was interesting, and the method of analysis novel. I thought members might also be interested, whatever their eventual conclusions, as the books were offered in our most recent batch of Early Reviewers.

I am sorry if anyone feels this is not in the spirit and tradition of LT, or that I have highlighted ideas or members I should not. It certainly is not my intention to present the interview as LT's position on the literature of the British Renaissance. In case it needs saying, we don't have one. Or if we do, it has not been communicated to me. I simply saw an unusual idea and an unconventional approach, and thought it might be of interest.

All that being said, I also want to acknowledge that it is the right of every LT member to disagree with both the interview's conclusions and the decision to post it. I mention this because the idea of free expression has been touched on in the thread. I do not consider it an insult to me personally, or to the site, that the decision to post the interview should be criticized, or even lampooned; nor do I consider it an insult to the author that her study should be rejected, criticized, or even ridiculed. While we want to promote interesting and constructive discussion, people have the right to disagree and to voice that disagreement, just as the author has a right to defend her thesis. I don't think anyone is best served by assuming malicious intention, however strong the disagreement.

Regarding the issue of self-promotion/spam/censorship, LT does have specific rules in this regards, and posts that have received multiple flags will in fact be deleted, as mentioned above. Specific groups also have their own specific rules, and these rules should be respected.

Anyone wishing to discuss this with me privately is welcome to do so, through the usual channels: leave a private comment on my profile page (https://www.librarything.com/profile/AbigailAdams26), or email me at: abigailadams@librarything.com.

Dr. Factorovich, I will be emailing you shortly.

According to one source (Wikipedia):

The English Renaissance was a cultural and artistic movement in England from the early 16th century to the early 17th century.

How does the work normally attributed to the Brontë sisters ("The Brontës were a nineteenth-century literary family", from another Wikipedia page) get labeled as works of the English Renaissance. Or has the definition been greatly expanded and I did not notice?

I grant the failure to find accurate information on some Wikipedia pages, like "Edward Stratemeyer", but find these two 2to be well cited and probably accurate.

James

>64 thorold: Volumes 1-2 of my Re-Attribution series have 698 pages that provide "extraordinary evidence" of my re-attributions of the 284 Renaissance texts I tested. I quoted a segment out of my "Brontes" article that also addresses several other attribution mysteries. This "Brontes" article will be published this spring or a bit later in the Journal of Information Ethics, and then you can read the full set of evidence to see if you want to counter it. The Re-Attribution series is already available for sale on Amazon, and I have given away free copies of it to Early Reviewers on LibraryThing. If the games the three sisters were playing altered their writing styles to be similar, then they would have all registered a single style, but instead there are two styles between the three sisters. There are intersections between these two styles that also hint at some degree of collaboration, but there is definitely no third style in this mix.

>65 AbigailAdams26: Dear Abigail: Thank you for your rational response to the points raised in this discussion. I have not asked for anybody to be censored from this forum, even if their comments have offended me. I do not think this discussion has gone uniquely badly, but rather it is indeed very normal for the types of discourse history-changing research faces in the public in our modern world. I have not reposted any posts in any forum where my posts were deleted, and I am certain I did not violate "spam" rules when I made the post being referred to. The reason my post was flagged was obviously because this group disagreed with my findings and not because it was self-promoting, as this is what they have repeatedly stated in this discussion without quoting the post that they deleted to let it speak for itself. I hope anybody who wants to continue this important discussion will just address the evidence and claims I raise in my studies. I welcome all relevant criticism because similar questions can occur to other readers who are only reading these excerpts I am mentioning in the interview/these replies, without the rest of my studies. I am happy to explain my findings further, and this is what I hope the rest of this forum discussion will be about.

>66 Keeline: As I believe I mentioned before, I have applied the computational-linguistic authorial-attribution method I invented to texts between 1560 through around 1940 in different projects. The Renaissance Re-Attribution series is the set of 14 volumes (2,500 pages) that I have just published and was available for Early Reviewer reviews - in it, I tested texts between 1560 and 1662. I also wrote (but haven't published yet) a book (300,000-words) on the Re-Attribution of the British 18th century, including the Daniel Defoe byline. And I have written this forthcoming article on the "Brontes" for the Journal of Information Ethics that covers British-American texts between around 1850-1940. I have tested the method on different periods and authors deliberately to check that it is always accurate in its attributions. Let me know if I can clarify this further.

>38 faktorovich: You are focusing on censoring my research without addressing what my research is.

Hardly. I'm just sitting here with a well-buttered barrel of popcorn. I have no interest in censoring you, and you can't compel me to take your intellectual product seriously when I consider it risible.

>65 AbigailAdams26: Thank you for clarifying.

Wow, this thread really blew up. But I'm posting in an epic thread, so I've got that going for me, which is nice.

>29 faktorovich:

"The Hebrew language had a significant early impact on many world languages. And Sanskrit has been acknowledged to be an ancient language that also influenced those that followed it. What does any of this have to do with my research?"

The cranks I refer to posit that Hebrew / Sanskrit / Tamil / Serbian /... is the mother tongue to Latin, English, Welsh, Chinese, Quechua... Not "related to", not "has had an influence on", but the ancestor to! This just cannot be true based on everything we know about these languages and their history; and, furthermore, these cranks make their claims based on crappy methodology and motivated by unscientific concerns. "What does any of this have to do with my research?" Well, the very first words of my comment were "This is the literary equivalent of those linguistic conspiracy theorists who ...". Your work is strongly reminiscent of those crackpots because it uses the same kind of argumentation (a complete up-ending of established research based on a single person's idiosyncratic methodology) and the same lines of defense. I recognize the flags, is all.

"You seem to have concluded ironically that the establishment is made up of idiots who have had a blind-spot for the British Renaissance as they have been propagating the greatness of "Shakespeare" via his pro-monarchy plays. "

Don't put words in my mouth. I never said anything of the sort. Don't create strawmen.

"Yes, ghostwriters are the downtrodden minority, who have received little credit"

Sure. I'm broadly sympathetic to that claim. My issue lies with the embedding of this concern within extraordinary claims and uncritical acceptance of the results of some self-published non-peer-reviewed stylometric pipeline. Your desire to highlight the plight of ghostwriters and to demand respect for their unrecognized work does not mean your wide-ranging rewriting of literary history is correct. In fact, wanting a result to go one way should be an extra cause for concern if the results indeed do go that way.

"Those who were refuting Einstein's theory of special relativity..."

You're not Einstein. This is an exaggerated comparison that makes you look like a victim, like an unrecognized genius who was eventually proven right. It's a flattering image that doesn't alter the truth level of your hypotheses, merely the appearances.

"reading the research they are refuting? You have not even mentioned the computational-linguistic attribution steps I included in the interview"

Please provide me with the doi of the paper in which you've published your algorithm and I will! I have full access to two well-stocked university libraries with all their journal subscriptions and to ILL.
Please don't ask me to pay for your self-published book. That's not what a proper scholar would do.
If I were to discuss the "computational-linguistic attribution steps" in the interview -- which are likely to be incomplete and abridged and dumbed down etc. for public consumption -- I'd likely be confronted with "well, it's better explained in my book anyway". So no. Give me a doi or let this go.
Remember: you're the one making the (extraordinary) claims. It's up to you to prove them right. Not up to me (or other members here, or literary scholars and digital humanists) to fail to disprove them.

"There are several typos in the Callaway article you cite"

1) Irrelevant to the results. 2) A blog post has different standards than scholarly books. 3) Many reviews of your works here on LT mention poor editing and spelling errors. I'm not holding that against you either. Spelling is totally irrelevant. Honestly.

"My explanation is far more sophisticating and logical than what Callaway does and it addresses the rules of storytelling in the literature field, instead of using NSC classifications that are more typically applied to the statistical study of cell behavior in biology."

More sophisticated also means more places where things can break. Look, the relative sophistication is not an issue here. I brought up Callaway's "fun experiment" (as I called it) as an example of the kinds of neat things computational methods can bring to stylistics. Also, to those of us working in digital humanities and linguistics and stemmatology it is absolutely bog-standard to alter phylogenetic models from biology to model cultural and textual evolution. That really isn't a weird thing to do, like at all.

"You seem to be claiming that Callaway is superior simply because scholars have agreed he is superior, and anybody who questions his superiority is not worthy of consideration."

Nowhere have I said this. This inference is entirely yours. Again, don't put words into my mouth. No strawmen, please. Plus, it sounds like you're reaching for that unfairly maligned victim role again.

(Callaway is a she, btw.)

"Instead of saying nonsense about how I walk, look or talk (which you cannot see given our distance and the textual method of communication), why don't you actually point out what specific elements of my findings you find to be quacking?"

DO NOT PUT WORDS INTO MY MOUTH. There's an idiom in English that goes "If it looks like a duck, and walks like a duck, and quacks like a duck, chances are it is a duck". It roughly means that if something (or someone) gives off all the signs of being an X, people are justified in calling that thing an X, even though looks may be deceiving, even though you shouldn't judge a book by its cover. Actions speak louder than words.

So. I did not say you look or walk like a quack, I said "Faktorovich's work, in contrast, looks, walks and talks like quackery" (emphasis added). That means, your work gives off all the signs of being unscientific poppycock, and so that's exactly what I'll call it. If your peer-reviewed algorithm convinces me otherwise, I'll retract this.

"anti-Semitism, nonsense and insults against a woman's looks"

DO NOT PUT WORDS INTO MY MOUTH! Don't try and paint me with filthy brushes to make me look bad. Or in other words: no strawmen, please. You created several strawmen in your post in your haste / eagerness to paint yourself as the unfairly maligned victim, the soon-to-be-vindicated Genius, the righteous Defender of the Downtrodden. This is not how a proper scholar does it.

I will no longer respond to strawmanning and victim-playing. Give me the doi's I asked for, and leave me to read your work at my convenience.

>36 faktorovich: You're calling me the same as the Roman Inquisition because I said I don't think LibraryThing should promote your self-published research without a second opinion? I work in research ethics and I assure you we do not execute people for not getting a peer review. We just don't let them publish their work in reputable journals until they do.

>72 Petroglyph: If you are going to dismiss any researcher as a "crank"; you have to at least cite this researchers name and the name of the article/book where this research appeared, so that I and everybody else can look up the proposed theory to test if it is indeed absurd or if there is some rational basis for this argument. It is irrational to dismiss any theory simply because you don't believe in it immediately upon reading a gist of what it is about.

A "conspiracy" can be the simple act of plotting or conspiring in any endeavor; some conspiracies are irrational, while others are entirely rational/ scientific or have a specific political goal that can be either achieved or not. You are using the term as if it is a fact that all conspiracies are false and fraudulent, but the American Revolution started and ended as a conspiracy. The main point I find revealing in this paragraph is: "a complete up-ending of established research based on a single person's idiosyncratic methodology". I mentioned Galileo before - he was alone in his conclusions about the relationship between the Earth and the Sun. His method was just as simple as my own - he looked at the sky and studied the movements of the planets and the sun. I am simply applying a group of 27 different linguistic tests in a systematic manner that reveals the quantitative similarities and differences between the linguistic styles of authors. Galileo would have had a calm career in academia if he had ignored the facts he saw in the sky, and just agreed with the established theory; instead he spent most of the rest of his life in confinement, and his ideas are still not fully accepted by some in our modern times who have argued the Earth is flat. If I had argued that I found strange numbers written on the back of my head, or had seen a dream about the Renaissance; you would be justified to call my findings idiotic. However, the data on the GitHub page I cited applies 27 tests X to 284 texts with X millions of words and compares all of these texts to all of the other texts. There are tables that explain the structural patterns, and tables that cite the original publishers/ performance information. This is the largest study of its type to be applied to the Renaissance; it responds to previous computational-linguistic methods and explains why these earlier attempts failed. Have you tried following the steps I provided? What about these simple steps is anything but simple statistical mathematics that are as irrefutable as 2+ 2? There are moments in scholarship when old ideas are proven to be wrong by new researchers; dismissing all conclusions that contradict established history/science is far more conspiratorial than allowing the math speak for itself.

While you are misinterpreting my statements, I have summarized your points precisely; while you did not use the word "idiots", it was implied.

Here is how scholarly dialogue is supposed to work: 1. Scholar A proposes an idea that changes the history of Britain, so it is published in a scholarly journal that welcomes such dialogue, 2. Scholar B reads the full book/article Scholar A has published; and then does additional research and publishes an article that refutes or confirms Scholar A's findings. 3. Then, there is a rebuttal or contributions from other scholars that debate the evidence to weigh if it is conclusive on either side. Instead, in modern academia: 1. Scholar A proposes a history-changing idea that is immediately and irrationally rejected from all "insider" scholarly journals and publishers by Scholar B, who send insults and nonsensical reasons for the rejection that show they did not read the research in question, and are simply rejecting it because they have noticed it proves their own research to be wrong. 2. Scholar A has to either trash their findings, or self-publish them. 3. If the work is self-published, all of the major review journals that are read by librarians who make purchasing decisions reject the work simply because it is self-published without even opening the book. So we are now at this final stage. And you are seriously arguing that this system is right to block all new and history/science-changing research from reaching the public?

It was not at all my goal to discover that ghostwriters were behind "Shakespeare", as I had hoped this was a real person and most of the bylines were accurate. The numbers just proved this intuitive desire in me to be wrong.

I am indeed not either Einstein or Galileo. I have not been arguing that I am a man from previous centuries. If I was Einstein, would you have similarly responded to his pre-fame papers by insisting that since he is not Galileo and has not already been accepted as a great scientist, his theories are not worthy of your research before you dismiss them as irrational?

I would be delighted to send a free review copy of the entire series to you via email - if you give me your address. You can request it by emailing me at director@anaphoraliterary.com. I am actively seeking reviews on LibraryThing, so your willingness to review the research is appreciated. Volumes 1-2 are 698 pages long and you have to read this full book before judging my method, as the entire thing provides different types of evidence that re-affirm the central linguistic findings. The simple steps involved in the method are included in the interview that this string is about: https://blog.librarything.com/2021/12/an-interview-with-scholar-anna-faktorovich.... I did deliberately simplify the steps for this interview to allow all members of the public to use this method without reading the full book to solve mysteries of their own.

The reviews of my work on LibraryThing have been libelous as there is no spelling/grammatical etc. errors in my books. They never cite any errors, and instead just use broad insulting language to dismiss my books as unworthy of public consumption. This is similar to the unspecific dismissals of my research in this conversation. I have not filed complaints about the falsehoods in these reviews because I believe all speech should be heard even if it is false, harmful to somebody being attacked and malicious. In contrast the Callaway article includes absurdly obvious typos like parenthesis () without anything enclosed in them, so these errors are absolutely unacceptable for any posted text in a blog or elsewhere. My knowledge of English spelling and grammar would become apparent if you read Volumes 3-14 of my Re-Attribution series that include extremely detailed notes on language uses to assist readers with understanding my translations from Early Modern English to Modern English. I have written thousands of words in this discussion today, so anybody who is unfamiliar with my work can also check what I have written here to see if I have made any mistakes even when speed-writing on a blog.

It is in fact common for computational-linguists to adopt biological or other hard-sciences approaches to testing the authorship of literary texts. My study explains that this is a mistake. The science of literary composition has very different rules from how nature makes biological cells.

If you did not make claims of quackery before actually reading even the interview where I summarize the method; you would not have to retract your unsupported opinion when it is proven wrong. Yes, you can take your time with the review of my work, but I am assuming you have given yourself an out by stating that you will not consider my research unless other scholars have already stamped it with agreement; as we discussed previously I have indeed self-published this series, so obviously it did not go through peer-review by this simple fact. Victim-playing? Imagine a scenario where we were talking about a rape or an attempted murder, and you were insisting that the victim speaking up about an attack was a case of "victim-playing". The statements made in this discussion are public-record, so let others decide if I am a victim or not. Don't divert the discussion into the irrelevant topic of my victimhood, and instead just read my research and respond to it rationally in a review that actually shows you are familiar with what you are critiquing.

>65 AbigailAdams26:

Abby, I have a couple of questions for you.

1. How does LT choose which authors are interviewed?

2. This book has been described throughout this thread as "self-published". Indeed, the author herself says that it is. (See >74 faktorovich:: "as we discussed previously I have indeed self-published this series,"). Yet the ER rules clearly state that "Early Reviewers limits participation to select publishers." So how does her admittedly self-published book come to be an ER offering? Was an exception made? If so, why?

>75 lilithcat:, re: >65 AbigailAdams26:

Indeed, other self-published authors are curious!

>21 faktorovich:
It's a British idiom for 'a lot of nonsense'.

Wow. The author's comments here would thoroughly put me off reading her work even if interested in the topic. Inability to hold a reasonable discussion and resorting to emotional claptrap instead is not exactly something that recommends her scholarly work.

I'm for the first time ever sorely disappointed in LibraryThing's staff. What's next? Cold Fusion? Flat Earth?

>74 faktorovich: I mentioned Galileo before - he was alone in his conclusions about the relationship between the Earth and the Sun.

Except he wasn't alone and incdentally he wasn't first either. There were quite a number of people in expounded a heliocentric view of the universe.

>75 lilithcat: I am an author-publisher. I own and direct Anaphora Literary Press, which has published over 300 different titles by different authors, dozens of which have been offered through Early Reviewers on LibraryThing. It is thus both true that I have self-published and that I have been published by a publishing company. I explain the fact that most of the best authors (Dickens, Twain etc.) published themselves with their own publishing companies in the "History of British and American Author-Publishers". The rule against self-publication is intended for those without publishing experience who are more likely to leave mistakes and create unprofessional books. When this rule is used by reviewers (as I have seen at many major newspapers/magazines) to bar all small-publishers, all self-published authors, and author-publishers, it bars access to the press without considering the quality of the texts being pitched.

>75 lilithcat: Your first question is a little difficult to answer, as I have only participated in a few interviews thus far, and don't have a large sample size. I would say the best answer would be that there is no set formula for the choice. That being said, it is at least partly driven by books or projects the LT staff find interesting, by what is current and notable in the book world, and by who we know or are acquainted with. We don't limit our interviewees to authors, but also aim to interview booksellers (we have one such interview forthcoming), publishers, and other figures from the book industry. If you have specific interviewees you would like to see, by all means send them along to me, although I cannot guarantee that they will ever appear on the blog.

In terms of this specific choice, I was approached by Dr. Faktorovich about the possibility of an interview, and being aware that she had contributed thirteen titles to our recent ER batch, and that these books represented a major project she had undertaken, thought it might be an interesting topic. I was intrigued, partially because the idea of this kind of computational analysis of literature was foreign to me, and partially because I appreciates that the project was making hitherto inaccessible texts (the works of William Percy, as well as those anonymous works attributed to him by Dr. Faktorovich) available to the reader of today. I have sympathy for such projects—I greatly admire The Other Voice in Early Modern Europe series, for instance—and as the only person in my masters cohort who worked with older texts—I wrote my thesis on three centuries of Reynard retellings for children in the Anglophone world—I appreciate the kind of archival research it must represent.

That being said, no interview presented on our blog, or excerpted in SOTT, represents LibraryThing's view on said subject, nor should our presenting of any interview be read as endorsement. We try to pick subjects that are of interest to readers, that is all.

2. Policing the border between publishing and self-publishing is actually rather complicated. Our policy, as explained to me when I started, was that books would not be eligible for ER if they were self-published, whether through some kind of pay to print company, or through a vanity press which only deals with one author's work (publishers set up by individual authors to publish their works alone). Small and independent publishers are eligible for ER if they print the work of more than one author. Dr. Faktorovich's publishing house, Anaphora Literary Press, falls into this latter category, and has been participating in ER for some years now, I believe.

I think the confusion here lies with the use of the term "self-published." In a sense, these books are self-published, as the publisher is one owned (I presume?) and run by the author herself. But they are not self-published, as we define the term for the purposes of ER. I hope this clarifies matters.

>80 andyl: I am certain that Galileo was indeed alone in his findings in his lifetime, even though such theories were previously vaguely proposed without the needed scientific proof to make them believable. If as you claim, he was not alone and there were others helping his position; then, obviously I am at a far greater disadvantage than Galileo, as I am indeed alone in my findings, and nobody else has yet read far enough into my research to join my view of British history.

>82 AbigailAdams26:

Thank you for the clarification re: "self-published".

>82 AbigailAdams26: Dear Abigail: Thank you for your kind remarks, and for the thorough explanation of your policy. Few newspapers/review magazines etc. I have pitched this project to for review have even noticed Volumes 3-14 that translate previously inaccessible texts; instead, they tend to stop at the idea that six ghostwriters wrote the Renaissance and dismiss the project without reading the details of how I arrived at this conclusion. While most readers in the general public assume that the British Renaissance has already been over-translated and over-analyzed, specialists in this field learn that the most historically interesting texts from these decades are inaccessible because they are handwritten, transcribed in old-spelling, or published with few and frequently inaccurate translations. I primarily added the translations to this series because they allowed me to find a few new pieces of evidence to confirm my six-ghostwriters conclusion in every page of annotations, but as I read the translated versions of these texts, I realized these are great literary treasures that are more deserving of public attention than canonical "Shakespeare" texts.

>49 faktorovich: "My study proves beyond-doubt"

Right there you reveal yourself as a schlock scholar. You're embarrassing yourself with all of the replies at this point. For your own self-respect, you should stop.

>66 Keeline: "How does the work normally attributed to the Brontë sisters ("The Brontës were a nineteenth-century literary family", from another Wikipedia page) get labeled as works of the English Renaissance?"

Don't you see? Despite volumes of letters, diaries, manuscripts from publishers, friends, the authors themselves, the use or lack thereof of the simple phrase "I can't" proves that two guys wrote everything from the English Renaissance, including those previously assigned to those canonical Renaissance authors, the 19th-century Brontes. /s

This is so ridiculous. As >64 thorold: said, extraordinary claims require similar evidence. I'd be embarrassed if one of my first-year students advanced such a thesis & defended it as you are. You must have taught comp. Go look at one of your old books and remind yourself how to argue a point honestly & convincingly. You're showing absolute contempt for this audience. Don't be surprised if they return it.

>86 susanbooks: "Beyond-doubt" is a legal term: "Beyond a Reasonable Doubt: The standard that must be met by the prosecution's evidence in a criminal prosecution: that no other logical explanation can be derived from the facts except that the defendant committed the crime, thereby overcoming the presumption that a person is innocent until proven guilty. If the jurors or judge have no doubt as to the defendant's guilt, or if their only doubts are unreasonable doubts, then the prosecutor has proven the defendant's guilt beyond a reasonable doubt and the defendant should be pronounced guilty. The term connotes that evidence establishes a particular point to a moral certainty and that it is beyond dispute that any reasonable alternative is possible. It does not mean that no doubt exists as to the accused's guilt, but only that no Reasonable Doubt is possible from the evidence presented" (West's Encyclopedia of American Law). The 698 pages in Volumes 1-2 and the annotations and introductory explanations in Volumes 3-14 of my British Renaissance series jointly meet this standard. The evidence I provide shows that the handwriting styles match the linguistic styles of the ghostwriters. I have also provided legal documents that support the assertions I am making from these decades. And various types of forensic accounting and biographical timelines that further strengthen the re-attributions. Anybody who actually reads this series' facts will be convinced that "no other logical explanation can be derived" than that these six ghostwriters committed fraud by using multiple pseudonyms, and generally wrote if not all of the texts from the British Renaissance, then certainly all 284 of the tested texts. The reader as juror/judge would be left without "doubt", with the exception of "unreasonable doubts" (including all of the intuitive comments that have been made in this discussion regarding folks having blind faith in "Shakespeare" being a real author). I have written extensive chapters on why these six specific ghostwriters are the only ones who had motive, opportunity, and had lifetimes (birth/death dates) that fit the timeline of their published texts. I have considered hundreds of other alternative bylines as potential ghostwriters, testing 104 of these bylines that were the most likely culprits given their biographies, and have excluded all but the six ghostwriters as the culprits. If this response is "schlock" (meaning: inferior); you have an ironic sense for this term.

'Beyond doubt' is not the same thing as the legal 'beyond reasonable doubt'.

This also has nothing to do with cataloguing, as that is based on what is printed in the book. A librarian may make a note of outside information, but catalogues what is printed in the book.

>83 faktorovich: oh this hurts my physics head. Nicholas Copernicus proposed and published his heliocentric theory in the year Galileo was 21. Galileo and Kepler both worked to prove and support Copernicus. His book was so important with astronomical data that the Catholic Church wouldn’t ban it because they used his data for calendars.
In your analogy, you are waiting for a Galileo to further support your thesis. Galileo never gets the credit for the heliocentric theory.

>87 susanbooks: You have not read my response to the nonsensical point about the Brontes Keeline has made. As I explained before, the excerpt about the Brontes was from my separate essay on texts from 1850-1940 and not from the Renaissance series that this interview with LibraryThing was about. The journal article had a word-limit so I did not have the word-space to explore if handwriting analysis would match my linguistic findings. Now that you mention this point, I looked into it briefly by reading a Guardian article on this topic. The male names that initially appeared on the "Bronte" novels were “Currer”, “Ellis” and “Acton Bell”. Curiously, Arthur Bell Nicholls (Charlotte Bronte's widower) was named as the person who sold "Brontes'" manuscripts in 1895 to a "literary forger" called Thomas James Wise. The echo between Nicholls' middle-name and the three initial male names' last name is not likely to be accidental. The sale of these documents to a forger makes it very likely that the documents did not actually exist prior to 1895, and were instead created by the forger to strengthen the case for the actual authenticity of the Bronte sisters as authors. As the surviving husband of one of the Brontes, Arthur Bell Nicholls would have seen a fiscal benefit from promoting their authenticity and otherwise marketing them as great writers. As Sotheby's has explained, the manuscripts attributed to "Emily"/"Charlotte" that came through or from the forger's hands then disappeared from public view until near the present moment, when they are finally up for sale, but have not been digitized into public-access to allow all scholars to evaluate their handwritings in comparison with both Arthur Bell Nicholls and Wise's handwriting styles. There are only a few scribbles that are in this collection from "Emily", and certainly not enough evidence to support the previous volume of shorter writing projects she would have had to do to become the novelist she is claimed to have become. The use of "I can't" is only one piece of evidence; the bulk of the evidence is in the linguistic-data file that I linked to in GitHub that you did not open.

>90 raidergirl3: You are at the same time not saying anything new, and what you are saying is nonsensical. As I stated before, there were "heliocentric" theories proposed before (one of them by Copernicus), but they did not provide the mathematical/planetary evidence that Galileo provided in his self-published book that moved this theory into being a mathematical fact proven beyond reasonable doubt. Kepler lived in Germany, while Galileo lived in Italy. Kepler is known to have sent a book to Galileo, but the two did not work together. The Catholic Church only exonerated Galileo from his condemnation on heresy charges in 1992, so they very much did indeed ban Galileo's book. Even if the Church used Galileo's data to update their calendars in his lifetime, this did not contradict their insistence on imprisoning him for his research.

>92 faktorovich: I didn’t mean to imply that Galileo and Kepler worked together; they were simultaneously proving the theory. I’d argue Kepler’s work did more to mathematically prove heliocentricity.
I used a pronoun instead of the name - it was Coperinus’s book which wasn’t banned due to its valuable data.
We just disagree on Galileo’s role in this. And since the church didn't exonerate Galileo til 1992, I guess his theory wasn’t proven to everyone beyond a reasonable doubt.

>93 raidergirl3: Galileo's theory was indeed proven beyond reasonable doubt. The problem was that the Church had unreasonable doubt that was rooted in its theological "history" of the universe. Because the Church was the judge/juror over Galileo's potential posthumous exoneration, they were not obligated to exonerate him as long as they had the power not to do so. Insisting the Earth was the center of the universe was more important to the Church than the fair execution of justice, or they did not see their faith-based belief in Earth's centrality as unreasonable until 1992.

11> 2Callaway's fun experiment in untangling the two authors of Good Omens is a good example of that"
Fun - thanks for sharing that!

My very 'wet' science building has a 'dry' team doing automatic analysis of journal articles: not so much looking at authorship but trying to parse meaning and collate systematic reviews.

>90 raidergirl3:
Yep and that doesn't include Thomas Digges who in A Perfit Description of the Caelestiall Orbes went a bit further than Copernicus. Other astronomers could also be added to the list who helped develop the heliocentric model such as Michael Maestlin.

It is also worth noting that Galileo did not prove his theory beyond reasonable doubt, because his theory was wrong. He championed a Copernican model with perfectly circular orbits, a theory that did not fully fit even the facts observable at the time, and certainly not the facts we can observe today. The Church had scientific consensus on its side in rejecting his model. Of the many different models under discussion at the time, it was the Keplerian, with elliptical orbits, that turned out to be most correct, but that was not what Galileo argued, nor was it obviously the right one with the evidence available in the early 1600s.

>97 ErlendSkjelten: A theory is a system of ideas designed to prove something. In a healthy scientific scholarship environment, all theories must be updated, revised and corrected in each generation that follows the initial theory design. With millions of scientists working across the world today, and thousands in this narrow field, it would be very strange if they did not find at least one idea in any 400-year-old theory that was erroneous and needed to be changed to fit new data, analysis and scholarship. Galileo's approach to measuring the movement of the planets, ocean-tides, and various other scientific strategies was the component that made Galileo's book uniquely logical and scientifically convincing for readers. It was banned by the Church because this observation-based science threatened the Church's insistence that it instead knew the truth about the universe's creation through divine inspiration. The Church's "scientists" were theologians who retold the mythology from the Bible about Earth's centrality without observing the facts about the universe's nature. "Consensus" just means agreement; a group can all agree to state that they believe in a religion, or to believe in the Earth being flat, or to believe in anything else; just by repeating this agreed-on point, the group has consensus. Thus, the presence of consensus in a scientific, theological or any other type of group does nothing to prove the thing being agreed on; it simply means that anybody who disagrees with the thing agreed-on cannot join the group because membership requires consensus. Galileo's refusal to consent, and willingness not only to not join the scientific community of his day, but to take their errors on trial as he faced the legal system for his disbelief in the consensus is the martyr-step that pushed later generations of astronomers to be able to join Galileo's theory and to form a new group with this different consensus. Kepler was one of these astronomers who was helped by Galileo's rebellion because despite religious tensions (with Galileo's theory in-print) he retained a position as the imperial mathematician and was not prosecuted for his mathematical/astronomic conclusions.

>98 faktorovich: Wow, we're going to argue Galileo here? The Tychonic system seems like the most reasonable one with the knowledge available at the time; as Tycho Brahe pointed out, besides the Earth moving, the lack of any apparent parallax to the stars and their visual size would mean they were incredibly huge (far larger than the Sun) and unbelievably far away.

To continue the analogy, it's not enough to make a good case from one direction; you also have to address the sticking points of those who disagree. Personally, I find it all too neat. Six authors wrote everything under a bunch of different names? "The long tail" comes up quite often in modern discourse; no matter what the top end looked like, I'd expect many authors who wrote one or two books.

Where's the verification step? You don't seem to have ever run it over a known set of works and checked that it produces the expected output. Can it distinguish Conan by Robert E. Howard, L. Sprague de Camp, Andrew J. Offutt, Robert Jordan, Leonard Carpenter, etc.? It's easy to toss together a system that looks good and produces output you like, but it's much harder to produce a system that's correct and that you can show to be correct.

All this because "being right in the face of adversity" is apparently a transitive property when comparing yourself to Big Names. I hesitate to name a famous person from centuries ago who was proven very much wrong for fear of triggering more argumentative digressions and well-akshully.

I saw this on twitter yesterday and it seemed apropos:

"You know, it's not enough to say the opposite of what everyone else is saying to be Galileo Galilei. You also have to be right."

I also want to know how many false negatives the system gives when looking at books we know are by the same person. Can it distinguish between Iain Banks and Iain M Banks? What does it say if it can?

In the Good Omens example they compared it on single author works first - and it's presumably reproducible.

I can't think why three sisters who grew up writing stories for each other in a shared world might share similar writing styles and word usage.

>99 prosfilaes: As I mentioned earlier in this discussion, I have proven the case for the six ghostwriters by providing not only computational-linguistic, but also other types of evidence from these decades in forensic accounting, handwriting etc. documents, in structural patterns between the texts, and hundreds of pages of other types of proof. I have also tested other corpuses to verify the attribution method works (18th-20th century). The "expected output" being judged to be the "correct" answer by previous computational-linguists is the reason they have been making errors in their re-attributions. If academia only judges a re-affirmation of the current bylines to be "correct", while any contradictory evidence is judged to be a faulty method; then, past attributions are repeated even if they were initially made intuitively and without any byline or biographical evidence to support them. Computational-linguists are thus pressured to alter their methods and data until the output repeats the original bylines with over 90% accuracy. Based on my re-checks of these methods, these computational-linguists never give the full method they are using or the full raw data set and the steps the data went through in their papers. When they do give this information, I have proven in various articles/chapters that they clearly are not reporting the actual data, or are only reporting the pieces that fit their primary-bias of needing to re-affirm the established bylines. In contrast, all of my data is available for free on GitHub, and I have summarized the basic steps anybody can use to check this data in this freely available interview with LibraryThing. The test you propose is irrational and impractical because: 1. Currently only books first-published before 1926 are in the public-domain, and the first book in the Conan series was published in 1967; so it is likely to be impossible to find the digitized full-text versions of enough of these texts to test them. 2. Given that these pop-fiction in a narrow genre and with a single storyline; my tests so far would suggest that it is very likely all of these stories were ghostwritten by a single ghostwriter (not necessarily anybody named in the bylines); so there is not likely to be any linguistic differentiation between Conan stories with the Howard byline vs. the Camp and Carter, or Howard and Carter byline. Therefore, even if you could create digitized testable versions of these texts, the answer linguistic testing would generate is not likely to be what you would want to hear. It is also possible there are indeed three or more different linguistic signatures in this mix, but only testing Conan texts would not be enough to verify if these signatures definitely belong to the authors in the bylines, as any or all of them could have purchased ghostwriting services. Thus, a fair and precise test would be to test most of the texts in this fantasy genre published across these decades by hundreds of different bylines, and then to compare biographies to the linguistic results. I have deliberately avoided authors not yet in the public domain or those who might still be alive, or whose editors might still be alive, as such testing is likely to become personal, instead of fixing a problem of past historical attributions that have already been subjects of scholarly re-attribution discussions. My tests are designed to be used by anybody in the public that wants to test any group of texts; because these tests are simple-mathematic calculations, obviously any biases I might have would be irrelevant to the group of texts and the results end-users come up with.

Therefore, if you have a desire to test if my conclusion about the six-ghostwriters is correct, just email me at director@anaphoraliterary.com and I will send back to you a free review link to the 14 volumes of this series and you will be able to see the verifying evidence I have already provided there, and then you could discuss with me any points that you might not agree with. Everybody interested in reviewing this series, is invited to email me for these pdf review copies.

More than half of the texts published between 1560 and 1650 were initially anonymous, but an absurd number of these anonymous texts have since been re-attributed by scholars to an array of different authorial bylines, or at times to people who never had their byline placed on any text during their lifetimes. As I explain in several chapters in the book, vagabond laws made independent authorship as a profession illegal; all authors had to be patronized by a feudal aristocrat who approved their employment. Publishing was controlled by a handful of official monopolies that were initially granted by Elizabeth I (including the one given to one of the ghostwriters, William Byrd, in music/poetry-publishing between 1575-1596, meaning that nobody could legally publish books in this field without Byrd's permission, and the monopoly was transferred to Byrd's affiliates afterwards). Elizabeth also established a duopoly in 1594, wherein only two troupes/theaters were legally authorized to perform plays in the city of London. The rest of my re-interpretation of historical facts from this period makes it very clear that the system was designed for monopolization by a few of these fields. Thus, the dominance of writing by six ghostwriters fits this legal framework. In contrast, given how little writers made and how few books were printed/sold in reality per-title; it is illogical to claim that any writer could have made a living as a "professional" writer by only publishing as few as a single book or staging a single play in a lifetime.

>101 Aquila:

I can't think why three sisters who grew up writing stories for each other in a shared world might share similar writing styles and word usage.

You win Talk today.

>101 Aquila: In the Renaissance with 284 tested texts, the texts with the bylines of the six ghostwriters in nearly-all cases matched the ghostwriters themselves. In the 18th century in a group of a few dozen texts, there were several authorial bylines that only matched texts with this given byline; thus confirming that when no collaboration/ghostwriting is involved, a clear computational match results. In the 19-20th century texts I tested there was too much collaboration/ghostwriting in this small group of 21 texts for any byline to only match itself, but in most cases texts by any given byline matched each other (they just also occasionally matched other bylines).

There is no difference I could find between Iain Banks and Iain M Banks, as this seems to be referring to the same person, who is also a recent writer, and thus his works are not yet in the public domain and difficult to access for quantitative testing.

The Good Omens method is not at all explained in that article, and the software that would have been used can be assumed to be very expensive and only accessible through special privileges to insiders. Thus, this Omens method is entirely non-reproducible. Whereas my method is reproducible by anybody in the public for free.

Just because you are imagining three sisters teaching each other at home with only one of them getting a basic education outside the home, does not mean that it really happened. Their biographies and writing samples could have been forged to appeal to female readers. Thus, computational-linguistics and scientific handwriting analysis is far more accurate in determining authorship in the Brontes' case than trusting the established biographical narrative without such unbiased scientific methods. My tests are for elements that are always divergent between writers because they come from unconscious preferences driven by character-type (extraverted/introverted), interruption patterns (period/commas), preference for exclamation or questioning etc. If three sisters could write on different topics; their research into these topics would have differed thus introducing word-choice/knowledge/stylistic divergences. Have you looked at the data I provide on GitHub? It is extremely consistent despite my application of thousands of checks as I compared 284 texts to each of the other 283 texts on 27 different tests. Again, if your theory about sisters sharing styles was correct; there would be only 1 style between the 3 and not 2 between the 3 as my data indicates.

>103 timspalding: What is this victory over "Talk" for the day based on? One example of siblings with different writing styles is Maria and Albert Einstein. They went through the same elementary school before their paths split apart. Similarly only one of the Bronte sisters pursued an education beyond their family's self-education. Maria also completed a dissertation at the University of Bern, but she did so in the literature field. Aside for Maria's dissertation, I could not find any books she published across her long lifetime. Brilliant authorship is a very rare trait in the human species, and it is very strange if three brilliant writers appear in the same family. And it is also very strange if all three of them develop identical "writing styles" without splitting into their unique fields of interest as Maria and Albert Einstein did. One example concerning the Brontes from my tests is that "Anne Bronte's" 2 tested novels both had exactly 12 exclamations per 100 sentences, obviously indicating a distinct writing style; but the novels of "Charlotte" and "Emily" varied between 8, 9 and 19 depending on their topics. And again the lexical density between "Anne's" novels is nearly-identical (45.94 and 45.14) whereas "Charlotte's" two novels vary far more widely between each other (47.6 and 51.54). The pattern also stands out in adverbs with "Anne's" near-identical (7.08 and 7.09) but the other three texts between the other two sisters are similar to each other but less so, indicating potential collaborative input from two stylistic hands. On the syllables-per-word test, "Emily's" Wuthering is identical to "Charlotte's" Jane at 1.43, and "Anne's" two novels are near-identical to each other (at 1.44 and 1.45), but the second "Charlotte" novel differs at 1.57. When these different tests are added together, it is clear that the two "Anne" novels were indeed both written predominantly by a single author, but the three "Emily" and "Charlotte" novels share a single dominant author between them with some collaborative assistance from another author or from a heavy-handed editor, who could have been the author behind the "Anne" byline. There are several other patterns that I did not discuss in my article like that "Anne" uses "as well as" as the 4th most-common phrase in both novels, and this phrase only appears in one of the other "Emily"/"Charlotte" texts in 5th place in "Charlotte's" Professor. When I have tested 18th century texts, there were many instances of these types of common phrases only appearing when an author wrote independently without collaborative input; in the 18th century, there were also several texts that had extremely high rates of near-identical linguistic measurements of this sort that again proved independent authorship, typically also by the same author claimed in the byline. This just wasn't the case in the Renaissance (where there was a lot of collaboration: re-combining of pieces of texts into new poetry collections/ versions of the earlier text or co-writing), nor was it the case in this short experiment into the 19-20th century that included the Brontes along with several other authors. If any of you would just glimpse at my data, perhaps you will finally start actually talking about my studies and not what you imagine them to be about.

Oh. My. God.

When will this end?

>102 faktorovich: The "expected output" being judged to be the "correct" answer by previous computational-linguists is the reason they have been making errors in their re-attributions.

So how do you test your model? Mathematical/computer models produce all sorts of wonky results, which is why anyone building one should have trusted test data to feed through the system to check the results.

Given that these pop-fiction in a narrow genre and with a single storyline; my tests so far would suggest that it is very likely all of these stories were ghostwritten by a single ghostwriter (not necessarily anybody named in the bylines); so there is not likely to be any linguistic differentiation between Conan stories with the Howard byline vs. the Camp and Carter, or Howard and Carter byline. Therefore, even if you could create digitized testable versions of these texts, the answer linguistic testing would generate is not likely to be what you would want to hear. It is also possible there are indeed three or more different linguistic signatures in this mix, but only testing Conan texts would not be enough to verify if these signatures definitely belong to the authors in the bylines, as any or all of them could have purchased ghostwriting services.

Robert Howard, L. Sprague de Camp, and Lin Carter are all well known for being writers whose income depended on their writing, and part of communities of writers who could have called them out for using ghostwriters. I don't know who Andrew J. Offutt and Leonard Carpenter are, which makes me wonder why anyone would want to put their name on the book instead of the real author. Looking it up, according to his son Andrew J. Offutt was pretending to be a science fiction author while he made his real money writing erotica under alternate names; that's someone who sells ghostwriting services, not someone who buys them.

because these tests are simple-mathematic calculations, obviously any biases I might have would be irrelevant to the group of texts and the results end-users come up with.

Thud. Thud. Thud. Okay, I've stopped hitting my head against the desk.

I have a simple mathematical calculation; does text 1 equal text 2? No, then they obviously have different authors. Boom, William Shakespeare's works were written by over 30 different authors, all under a collective author name. Or we can use the more sophisticated tools at https://translatedlabs.com/language-identifier: this tells us that most, if not all, of those 268 works are written by the author they give the pseudonym "English EN-GB". Which one is correct?

it is illogical to claim that any writer could have made a living as a "professional" writer by only publishing as few as a single book or staging a single play in a lifetime.

Okay? I didn't make that claim. I would assume that then, like now, many people tried their hands at writing, and at least some of those sold a book or play, or had enough money or fame to get their book or play published, even if it wasn't popular or successful enough that they wrote a second.

>106 susanbooks: Relax and pass the popcorn. I’m taking odds on how long this can last.

Has anyone done a comparison using the I Write Like website described here: https://en.wikipedia.org/wiki/I_Write_Like

I'm probably not the only one here who baulks at the amount of iffy (as perceived) premises mooted in this thread. Of course, as a non-expert I'm open to criticism that it's just my ignorance getting in the way--and perfectly willing to accept it when that's the case.

Some of the things that are unclear to me:

--how is "style" defined for the purposes of CL (computational linguistics) analysis?

--is CL analysis accompanied by other, traditional forms of analysis, such as, er, comprehensively reading the text? I don't follow the field of AI research much, but l think that it's still the case that computer programmes are better at counting words than understanding their meaning.

Another reason for asking this is that I was struck by this >91 faktorovich:

Now that you mention this point, I looked into it briefly by reading a Guardian article on this topic. The male names that initially appeared on the "Bronte" novels were “Currer”, “Ellis” and “Acton Bell”. Curiously, Arthur Bell Nicholls (Charlotte Bronte's widower) was named as the person who sold "Brontes'" manuscripts in 1895 to a "literary forger" called Thomas James Wise. The echo between Nicholls' middle-name and the three initial male names' last name is not likely to be accidental.

I may be misunderstanding (sort of hoping I am), but does this mean that until reading that Guardian article, the poster didn't know the information she copied over? If so, that seems a strange position from which to undertake this study, which after all goes to negate a great deal of the subjects' biography.

One of other confusing points is the assumption about shared style--that it's all or nothing, or that style doesn't change over time or may not change deliberately etc. (it's a whole host of questions really).

Related to that, I'm confused by the weight placed on such ubiquitous expressions as "I can't" and "I did not". I realise that these are just two of the six "markers" of style mentioned and that perhaps I'm not appreciating some synergistic result when all six markers are considered. Nevertheless, the fashion in which even just "I can't", for example, has been highlighted encourages me to wonder: since there are only two forms of this phrase, contracted and uncontracted, doesn't that from the get-go impose a terrific reduction on the number of "styles" that can be discerned based on it? Again, I realise there are a few more markers involved, but I don't know whether they are of significantly greater complexity.

Finally (I'm sorry for the length but I did try to limit the number of points to bring up!), while this proves nothing in itself, I was so struck by the similarity between this result and one instance from my own field that I think it's worth mentioning. Decades ago when sequencing whole genomes had taken off we had a lot of work calibrating methods for sequence comparisons. And I remember very well the excitement when we received some new software and started feeding it our data, looking for genes that fit our clones. Thing is, we started getting amazing hits right from the start--literally the first sampled genome turned out highly matched... and the second... and the third...

Long story short--at a sufficiently low level of discernment all DNA is the same, and ravens are no different to the writing desks.

The tale of hundreds of works previously attributed to dozens (or is it hundreds) authors getting reattributed to just six; and the three Bronte sisters getting squished into two (men at that), can't help but seem to me another example of absurd reductionism through computing.

>105 faktorovich: One example concerning the Brontes from my tests is that "Anne Bronte's" 2 tested novels both had exactly 12 exclamations per 100 sentences, obviously indicating a distinct writing style; but the novels of "Charlotte" and "Emily" varied between 8, 9 and 19 depending on their topics. ...

I enjoy a good data dive, trying to pull facts out of a morass of limited information. But I try not to be so cocky as to assume that my answers are the end all and be all of anything. My biggest problem with that argument is that it's ungrounded; show me Twain and Dickens and Bulwer-Lytton, and if your tests can separate them, then I'd find that at least a stronger argument. I suspect, however, that authors' styles vary enough to make such tests questionable.

>110 LolaWalser: **ding, ding, ding!**
Points to the lady. This digitized analysis appears to be the way the protagonist “reads” books. Have you seen how broad her subject matter of published opinions is? I’m not sure thoughtful reading of everything she claims to have read is physically possible without a time machine.

>104 faktorovich: "The Good Omens method is not at all explained in that article, and the software that would have been used can be assumed to be very expensive and only accessible through special privileges to insiders. Thus, this Omens method is entirely non-reproducible. Whereas my method is reproducible by anybody in the public for free."

False. Done in R, with free libraries. The blog post says "Using a training set of texts by Pratchett and Gaiman, I used the R package Stylo to analyze Good Omens. (Specifically rolling nsc classification with 50 features and 5000 words per slice)." Here is an in-depth write-up of the contents of this package (direct link to the pdf).

How can it be you are so unfamiliar with R that you "[assume] it is very expensive and only accessible through special privileges to insiders"? Like, seriously, how is it you are unfamiliar with R? That is an honest question, and I would appreciate a straightforward answer. No blame-shifting, no excuses, no comparisons to paid software packages or to your own. How is it you are unfamiliar with R?

You must see that your unthinking reaction of tarring a bog-standard programming language that is famous for being absolutely free and that is routinely used for stylometric analyses with unfounded accusations of "very expensive" and "only accessible through special privileges to insiders" is not a good look, right?

>104 faktorovich: "There is no difference I could find between Iain Banks and Iain M Banks, as this seems to be referring to the same person"

They are different pen names of the same individual, yes, who split his output (litfic and SF) across the middle-name-less and the middle-name-ful one, respectively. This is an irrelevant objection, though. You were asked in >101 Aquila: "I also want to know how many false negatives the system gives when looking at books we know are by the same person. Can it distinguish between Iain Banks and Iain M Banks? What does it say if it can?"

Well, can it? If the algorithm could, what would that mean? If the algorithm couldn't, what would that mean? You can't just assume that "There is no difference I could find" when you haven't even tried to analyze the bodies of work!

Really, we'd like to know if your algorithm is reliable, before we are even willing to accept your complete re-writing of multiple centuries of authorship attribution. So let's put it to the test: how does the algorithm fare when applied to the body of work of a single, undisputed author?

Furthermore, let's assume for a second that post-1926 English-language publications are off-limits.

Have you tried out your algorithm on, say, Ulysses? This text is readily available, in that good old plain-text format that is so very very suitable for machine reading. We also know that this text is by a single author, James Joyce. Uncritical applications of stylometrics to this text can produce the wrong results, namely that i was written by multiple different authors (I suppose you would call them ghost-writers). We know that that is incorrect: Joyce deliberately employed multiple styles throughout this book. Instead, this means that stylometric models can be wrong if they give massively unexpected results and must, therefore, be used judiciously.

How many authors does your algorithm say this text is by? Is the author different from the one(s) who wrote Dubliners? From A Portrait of the Artist as a Young Man? From Exiles? Is there a single author behind these texts? Multiple? How are they distributed? What does your algorithm say about the "contributions" of various ghost-writers (if any) across these texts?

If your algorithm finds multiple authors -- does this mean that multiple people ghost-wrote Ulysses? If your algorithm finds a single author -- how much leeway is there within a single author's voice for this to be the case?

One present-day author whose works we know beyond a reasonable doubt that you have direct non-copyrighted access to is Anna Faktorovich. Can you apply your algorithm to yourself? You've produced several volumes of both fiction and non-fiction, i.e. a varied body of work. If you apply your algorithm to your own works, what are the results? Are your works by the same person? How many ghost-writers (if any) are present in your own non-fiction? In your fiction?

>107 prosfilaes: As I explained even in my brief interview response, my method includes self-checks within the basic process because it includes 27 different linguistic tests or tests for different types of linguistic measurements. The combined table of matches vs. non-matches indicates which texts in the tested corpus are similar to each other and thus share a linguistic-signature, and which are different from these and form their own linguistic signature groups. Because there are 27 quantitative tests, at least around a dozen of them have to come up with the same attribution answer for a match between texts to be established; each of these tests is equivalent to a re-verification alternative method beyond applied to the same corpus. Once I analyze the data and separate the texts into linguistic-signatures; then, I do not trust any one of the bylines in a given group, but instead test all of them on their established biographical facts to consider which of them are the most likely potential ghostwriters/authentically-bylined-writer, vs. pseudonyms or another category. While at the start of this study of the Renaissance there were many uncertainties, all of these wonky elements were resolved by the time I finished writing the 698 page book, and expanded the corpus to 284 texts; you really have to read the whole book to see why the final conclusions are entirely un-wonky.

In the Conan case. One of the most difficult parts I found in reaching the final conclusions in my study was distinguishing the underlying ghostwriters from ghostwriting-contractors/pseudonyms etc. In case of most modern writers, it is extremely unlike that a pop byline can be a pseudonym without a real person behind it, or this information would have become public. It would also be difficult to trace the finances of any modern author (Andrew J. Offutt included) to determine if there is evidence there of money coming in or out related to ghostwriting services. This is precisely the type of information that I found to confirm my findings in Renaissance financial documents (such as "Henslowe's Diary") that have been digitized. My point about these Conan writers is that it would be far more difficult for any group of several writers to share a similar Conan style, and it would be very easy for a single ghostwriter to write the full series while selling bylines to multiple ghostwritten-byline purchasers. I certainly cannot guess without any research who the ghostwriter vs. ghostwriting-purchaser in this group were.

I tried to use your TranslatedLabs link to check one of the texts as an experiment, and I received the error message that I have reached my data-limit for the day without testing any previous texts; so this tool does not work. I have no idea what 268 texts you are referring to, or why TranslatedLabs has attributed around 268 texts to EN-GB. It is possible that these are the over 200-plays that Percy claimed to have written across his professional career under one of his pseudonyms.

This was a period when the English language mutated by multiplying in size from Middle English into Early Modern English. The Workshop (including Percy under bylines such as "Shakespeare") made up so many new words, or adopted so many words from foreign languages that their texts might have read like what they were a combination of multiple foreign languages to most of the public in England that was not yet taught this new language. It is thus absurd to imagine there were even the brightest students who managed to learn this enormous volume of new textual materials etc. to create an extremely specialized and formulaic play, and then never wrote or published again (since there are few closeted unpublished plays from these decades).

>113 Petroglyph: >>Like, seriously, how is it you are unfamiliar with R?

Reading through this entire thread, this is what I'm really stuck on! How is it even possible to do that much stats work and not know what R is? I too am super curious.

That's like, idk, claiming to have discovered evidence that the Romans discovered America and not knowing what Latin is.

>109 Taphophile13: I Write Like is not a scholarly attribution method. 1. Nowhere on this website is there any statement regarding the types of elements the site tests for to determine what a text is like. 2. Its Wikipedia page basically says it only tests for vocabulary. I tested one of my 284 old-spelling texts that uses Early Modern English vocabulary, and it determined that the text is like Agatha Christie. The result did not include any information about the words that were similar between this text and Christie. Thus, it is entirely possible the result are: 1. entirely random without any vocabulary or anything else actually being tested, or 2. the tests are just for a few rare words picked at random from famous authors like Christie. This tool almost appears to have been intentionally designed to prove computational-linguistic attribution is impossible because its results and its lack of stated methodology have rightfully led users to criticize that it leads to nonsensical results. Unlike this approach, my method is entirely transparent, reproducible, and leads to consistent results that accurately attribute authorship.

The source code for I Write Like is linked to the bottom of the input page, if anyone is curious. And >109 Taphophile13: yeah, I've played with multiple times over the years. The first time, I learned about authors whose work I have since enjoyed. It's a fun program.

Who else remembers the old gender guessing algorithms than were so frequently wrong for published writing?

That was one of the first things I thought of when this topic was introduced. That, along with all the authors who turn out to be a group of authors for each byline, along with certain types of online warriors who believe only men write "real literature", along with how an author can deliberately change style markers to match or contradict themself or other authors, along with much publishing editors used to determine style,....

>110 LolaWalser: You are right in questioning the definition of "style" in computational-linguistics because it has rarely been clearly defined. For the purposes of my study, I judged "style" based on 27 tests that measured: punctuation, lexical density, parts of speech, passive voice, characters and syllables per word, psychological word-choice, and patterns of the top-6 words and letters. Style can also refer to literary concepts such as metaphors, or variant social meanings of language, or various other things that are not quantifiable in a simple table that can compare hundreds of texts to each other in 1s and 0s. The number of different punctuation marks, passive voice and the other elements I measured can all be broken down into single-numbers (percentages or per-100 sentences etc.). Therefore, when I state that there are only 6 linguistic styles in the 284 texts I tested from the Renaissance, I am not referring to something vague and uncertain, but rather to the mathematic facts in these texts' data.

My method is accurate in contrast with previous approaches in a large part because I have combined the quantitative analysis with various other steps that analyze the structure and contents of these texts. One clue of the uniqueness of my approach is that I have translated 12 volumes of previously untranslated Renaissance books as part of this series to provide additional attribution proof in the annotations. No previous computational-linguist has also done any translation of the texts they are computing. As I mentioned earlier, I relied on publicly-available free software to count the number of punctuation marks etc. in these texts. The innovation in my method is combining 27 different tests, and adding various other verification approaches on top of these.

As I explained, my forthcoming Journal of Information Ethics article only briefly mentions the Brontes in a group of 21 different texts. The central argument in that article is disproving and showing the errors that were made in a previous computational-linguistics article on "Unmasking" by a group of researchers. I did not perform handwriting analysis on the Brontes handwritten texts because this would have added at least a few thousand words to the article, and I would have had to add similar handwriting studies for all of the other 18 texts, expanding this article into a book.

Percy's first plays and interludes that he published in his youth in 1584-5 match his latest plays such as "Captain Underwit" that was published a year after his death in 1649. While there are more formatting/ spelling errors and other glitches in his earliest experiments, his style (and that of all of the other authors I tested) does not change significantly enough across a career for these texts not to be clearly identifiable as of a single-style with my computational method.

The 3-word-phrases (the top 6 most-frequently-appearing of these out of all possible 3-word phrases in a given text) I pointed out as revealing in the Brontes' case are not one of the 27-quantitative tests involved in the basic method. I collected these 3-word-phrases for all texts and used them to find obvious patterns of phrases that only appear in the work of any given authorial-style. The use of a contracted vs uncontracted "I can't" is a significant stylistic divergence that is revealing when one of these appears among the top-6 most-common phrases because this means there are many instances of this preferred usage in the text vs. the alternatives. I did not use any 3-word-phrase patterns to establish attributions, but just used them in the writeups to show that the less easily understood mathematic matches vs. non-matches were also confirmed by these verbally-descriptive elements.

I can see how you perceive this to be the case, but I started working on this project around 3 years ago. I saw glitches like the ones you encountered first as I wrote the 300,000 words on the 18th century, and gradually I adjusted my method, and did more research until I identified the glitches and the strands of "DNA" were clarified and finely defined. And then I wrote the 698-pages in Volumes 1-2 of this study after expanding the corpus from around 100 to 284 texts. I expanded it not for the sake of checking more texts but because I needed more data to check potential glitches. There have not been any previous computational-linguistic studies of the British Renaissance that tested 284 texts, so my findings are the most precise and deeply thought-through results in this field. There are 104 bylines in my corpus that I am re-attributing to 6. I have proven the Renaissance 6-ghostwriters case in this series of 14 volumes, whereas I have only written a couple of pages about the Brontes. I don't know why you guys are more mesmerized by the magic trick of three women turning into two men, but I would have to test hundreds more texts and perform years more of research to reach the same level of certainty about the Brontes as I have already reached about the Renaissance. Even if the 6-ghostwriters theory is more difficult to believe of the two, it is the one for which the evidence I have collected makes it an established fact.

>109 Taphophile13:

It says I write like Dan Brown, which makes me sad.

>120 paradoxosalpha: So sorry. That would make me sad too. Perhaps a different test paragraph would help.

>109 Taphophile13: I tried it with three reviews I've posted here, and I got Kurt Vonnegut, David Foster Wallace, and Margaret Atwood.

>122 amanda4242: It seems you are very versatile.

>111 prosfilaes: Since you asked, I upload 3 texts by these authors from Project Gutenberg: Bulwer-Lytton - Godolphin; Dickens - Oliver Twist; and Twain - A Tramp Abroad. I ran them only through Analyze My Writing's Basic tests. There are obvious divergences that prove these are texts by different authors. 1. Punctuations: Bulwer (173 commas per 100 sentences), Dickens (191), Twain (137). There was a bit of similarity between Bulwer and Dickens on questions and exclamations, but they were still different from each other, and Twain was very different from both of them. But Dickens and Twain were more similar to each other on their use of semicolons, yet still were a point different. Even if I ran these 3 texts through all of the 27 tests the results would not be conclusive because at least 20 texts should be in a corpus for the 18% similarity vs 82% divergence rule to apply. While the differences between 10 question marks in Bulwer and 11 in Dickens seem small with only 3 texts in the mix, if 100 texts were measured, several of Dickens and Bulwer's texts might prove to have exactly 10 vs 11 measures, and this would confirm that these are distinct linguistic styles. The commas measurement is uniquely divergent between these 3, so if I only referred to it, it would make this simple test seem like it is enough to establish authorship. But it is too easy for a glitch or an abnormality to creep into any one of these tests that is corrected when the corpus is big enough and the full range of tests is applied, accompanied with additional verification approaches that take a few days to carry out. Let me know if you want me to make the full experiment on any group of texts, but there should be a logical reason for this experiment, other than curiosity. The data on my GitHub already establishes the accuracy of my method on two different types of groups of texts.

>116 faktorovich: my method ... leads to consistent results that accurately attribute authorship.

How do you know that? You've told us about the results that contradict what was previously believed; you've told us nothing about the results that establish that your method works.

>112 2wonderY: If you calculate the number of words I write per-hour and how much research I have done just to respond to this thread of texts, you will figure out that it is indeed possible for me to read about, research and to publish about the range of different subjects I have covered.

>121 Taphophile13:

While my published fiction is "like" Dan Brown (although not as remunerative as his), a sample from one of the book reviews I wrote last month is "like" H.P. Lovecraft, which is more satisfying.

>127 paradoxosalpha: That's a much better result. My own reviews are like Arthur C Clarke whom I haven't really read.

>113 Petroglyph: You have linked to an article about "R package Stylo" and not to a software that can be used to test texts. This article explains that this package uses basic n-gram or word and character analysis on texts. The most common words and characters are only 2 of the 27 tests that I applied to my corpus, so my tests are far more complex and thus accurate than this R approach. I only have access to TexShare in my rural region of Texas, as I do not have institutional support for my research; so I could not access this package even if it is available at some research universities. Most of the public is in a similar boat, and does not have access to these types of institutional resources, and thus cannot check this method for themselves. In contrast the tools I cite are publicly accessible from any computer with an internet connection for free. By "50 features" you mean you analyzed only 50 semi-frequent words out of these texts. I explain why only testing a few selected words starts any study with extreme bias that cannot lead to trustworthy results. What your write-up lacks is even the specific full list of the 50 features you have selected, why they were chosen, and the raw data you generated before you turned it into your seemingly neat graphs. The R programming language is not relevant to my analysis because the basic tools for counting the number of punctuation marks etc. have already been programmed and are available for the public to use. If I had used a programming language to test the texts I analyzed; then, I would have also not been able to give links to the public to free websites where they could double-check my results. Alternatively if I had developed my own website that allowed users to perform the 27 tests simultaneously, this would have taken me years that I have instead spent on writing the 14 volumes of this series; there was no need for me to reproduce existing simple linguistic counting programs like Analyze My Writing. My goal was establishing the correct attributions of the Renaissance, and not selling a computer program I designed.

You have explained that "Iain Banks and Iain M Banks" are 2 pen-names of the same person, so checking the texts by this author against each-other is as rational as checking 2 texts by any established byline if the author happened to use a middle initial in one of them. There is no mystery to solve here unless you are saying you doubt these two Banks are the same person; this can be the case if somebody else wanted to benefit from Banks' status and used a similar byline to fool readers into interpreting the texts as being by the famous guy.

Sure, I can test a single undisputed author. I would recommend testing Dickens or Twain, since both were author-publishers and I anticipate their works will show clearly a single authorial style, and both of them have written dozens of digitized texts that can be tested. Go ahead and name the specific challenge and I'll proceed with carrying out the experiment.

To find how many underlying authorial-styles are behind Ulysses, I would have to test a few of Joyce's other texts and dozens of texts of other authors published in James' circles/ by the same publishers/ in the same genres. Intuitively the style in Ulysses is extremely different from "Dubliners" and "Portrait", so if my tests indicated different authors wrote these texts, it would not be unexpected from my perspective; if the styles do match, this would confirm that my method can identify an authorial style even if texts seem to be extremely different intuitively. You do not know if Joyce truly wrote Ulysses himself, or if he used different styles or if this was a text that combined contributions from multiple writers that were stitched together. The tests would only indicate which segments of Ulysses were by different writers if there were around 100 texts in the corpus that included all of the potential ghostwriters with multiple samples from each. To explore this question fully, I would have to write an entirely new book. So, instead of sending me on this Ulyssian journey, why don't you instead start by taking me up on the offer to read my finished Renaissance series for free by emailing me to request a copy at director@anaphoraliterary.com.

Sure, I can apply the tests to myself. You should know that I have done professional ghostwriting in the past, though I cannot disclose for whom. You would have to test millions of texts with around a million bylines to figure out what I have ghostwritten, as it is only a few projects in a sea of modern publishing. It would be less extreme for me to just test the texts with my own byline, but only testing my own tests against themselves would once again fail to establish what they are different from. For every text by me, I would have to add a few texts by other bylines writing in the present (and again I try to avoid testing anything published after 1926). Testing works by any single given author produces a set of numbers that represent the range of that author's style; only when this style is compared with other styles do these tests indicate authorial attributions with my method.

>115 melannen:
I don't think she does stats!

Well, averages and characters per words, yes. But I don't think she applies even t-tests or chi-square tests. How statistically significant are the adjectives-per-thousand-words between the ghost-writer behind one Brontë and the other?

Instead of using software and performing the tests herself (word counts, POS assignation, ...), she copy/pastes texts into online text analyzers, and then copy/pastes their output into a spreadsheet. And going by the interview, where she encourages people to copy her workflow, part of the data treatment consists of manually changing everything within 17-18% similarity for one text on a given test result to a 1, and everything else to a 0. And to do so until all the percentages have been changed into ones and zeros.

In other words, she's manufacturing the data. Changing percentages of similarities (as spit out by online generators) to either 1 (similar) or 0 (not similar at all). No wonder her results are so different from everyone else.

As I said back in message #11: Garbage In, Garbage Out.

>117 aspirit: I read through some of the source code. Christie is only mentioned in a long list of author-photographs. Most of the code includes messages uses receive while using the program. What is needed instead of all this programmer-speak is a simple set of linguistic rules that the program uses to establish what a given text is similar to.

>118 aspirit: I explain the gender bias that has led publishers to prefer men with linguistically-dense styles for mystery-writing, and women for linguistically-light styles for romance-writing in my previous book "Gender Bias in Mystery and Romance Novel Publishing".

>125 prosfilaes: I have explained that I researched handwriting analysis and documentary evidence that have confirmed my findings. You can see my articles where I include my handwriting findings on the series' page: https://anaphoraliterary.com/attribution. Once again, just ask me to send a review copy to you of the entire series, and you can see for yourself how I have confirmed my central computational-linguistic method with various other types of evidence.

>130 Petroglyph: But *I* do stats for fun by copy-pasting things spat out by online analyzers into spreadsheets and screwing around with them without ever quite understanding chi-square. Once I even proved that the Voynich manuscript was a map of Oak Island! But I at least know what R *is*, even if I'm too lazy to use it.

>129 faktorovich: R is free software. Anyone can download it and use it for free, anywhere in the world. It's available here: https://www.r-project.org

Stylo is a free package for R. It can also be downloaded and used for free, by anyone in the world. It's here: https://cran.r-project.org/web/packages/stylo/index.html

They're both fully documented, and are also GPL licensed, which means you can even look at the programming that makes them work, to make sure it's doing what it says it does.

They are a little bit less intuitive to use than tools like Excel, but for someone who is doing groundbreaking statistical research it's definitely worth the effort to use the many free online tools to learn how to use them!

>130 Petroglyph: I have a section in the Renaissance book where I test the mean results (t-tests/chi-square) in the data against outliers and check random statistics against the unique results I received in this study. I find in this section that my results cannot be accidental, but rather must indicate blatant linguistic distinctions between the authorial styles.

I don't think you understand what it means to run a counting-test on a text. It does not matter if I have invented my own exclamation-counting program or used a free public website to derive the answer - the results are the same - the system just counts the frequency. Obviously I have to copy the data from the software program into my spreadsheet if I need to run comparative tests on this data. I also don't think you understand that in computer languages "1" means something like "yes" (in this case "similar") and "0" is like "no" (or in this method "different"), so by changing the data into basic "similar" vs. "dissimilar" outputs based on their degree of similarity or difference, and then adding up the number of these "1" (similar) results, I establish the precise degree of similarity between each pair of texts in the corpus for a possible result between 0 and 27 or an enormous range of similarity variance. I am not manipulating data, but rather recording the 0 when texts do not fall within 17-18% of each other, and a 1 when they do fall within this range. Everybody else does not make the data available, but my GitHub page has all of this data, including the raw data from the tests with the different numbers and then you can see in other tables which of these numbers I have changed into 1s and 0s.

>134 melannen: Again, the R programming language is designed for folks to create statistical tools that count the number of commas etc. These programs have already been developed and are available for free online, so it is nonsensical for anybody to write new programs that repeat this completed labor. The Stylo package you linked to does not have any practical functionality accessible to the public that would actually allow for linguistic testing of texts to determine authorship. When I was researching what method I should use, I considered not only these tools by dozens of other tools advertised in previous computational-linguistic articles. After downloading a few of these programs and experimenting with them; I found most do not function properly, many are not really free available or have pay-walls to actually apply them. If you disagree, post a set of simple steps (in 1 paragraph) like the ones I gave for my method that the public can use to test their own texts with Stylo. If you cannot summarize the steps in 1 paragraph; it is not a method that can be tested without specialized knowledge, and thus it is only for insiders, and this makes it very easy for specialists in this field to manipulate data because their results cannot be checked or clearly understood even by advanced literature research specialists, and probably not even by computer programs in other fields. There is nothing uniquely special in using software that the public cannot use or understand. Excel or any spreadsheet is enough for my method; and because of this even a middle-schooler can use it to test if his classmates are buying papers from a paper-mill. Solving the intended authorship mysteries is the goal of my method and not making it sound unapproachably difficult and incomprehensible.

>135 faktorovich:
No, you change a degree of similarity into complete identity ("1" / yes) or completele difference ("0" / no), thereby obliterating the finer distinctions. What's your reason for enforcing this 17-18% cutoff where not similar (0) becomes similar (1)? Why those two figures? Why not 19? Are there cases where you count everything within 18% range as similar, and other cases where you cut things off at 17%? Why?

Adding up the ones to get to a similarity score presupposes that all of your 27 tests have identical weight. Surely, word density or sentence length are more telling than whether an i or an s is in the top six characters?

>136 faktorovich:
Oh my god. Stats are too hard for the average person, therefore her method, which completely sidesteps mathematical constraints designed to remove bias, is to be preferred.

"The Stylo package you linked to does not have any practical functionality accessible to the public that would actually allow for linguistic testing of texts to determine authorship."

That is exactly what that package was designed to do. To vastly superior degrees of sophistication than your handcrafted spreadsheets.

"If you disagree, post a set of simple steps (in 1 paragraph) like the ones I gave for my method that the public can use to test their own texts with Stylo. If you cannot summarize the steps in 1 paragraph; it is not a method that can be tested without specialized knowledge, and thus it is only for insiders, and this makes it very easy for specialists in this field to manipulate data because their results cannot be checked or clearly understood even by advanced literature research specialists, and probably not even by computer programs in other fields."

If you start from a position of basic distrust of specialists, then a position of distrust is exactly where you'll end up. Some of these methods rely on linear algebra to condense an n-dimensional space (say, for the n most frequent words in a corpus of texts) into a two- or three-dimensional scatterplot. One commonly used method to do this is principal component analysis (PCA).

If any of these words frighten you, you can't just throw together your own method and assume you'll get equivalent or better results than the ones specialists developed to handle multi-million word corpora and to avoid erroneous interpretations.

(edit: a word)

>134 melannen:
Nothing wrong with that! Tinkering with things teaches you about them. And at least you're not claiming to have decrypted the Voynich manuscript, nor are you demanding to be taken seriously.

>129 faktorovich:

Read the relevant portion of my message in >113 Petroglyph: again, but more slowly. R is free. Anyone who learns this programming language can use it. No institutional access is necessary. Just a computer. Which you have.

"I could not access this package even if it is available at some research universities"

Download R. It's free. Probably the GUI app Rstudio, too. That one is free too, and it makes things easier. Then, at the R command line, type this:

> install.packages("stylo")

Boom. done. Just like any other R package.

"basic n-gram or word and character analysis on texts"

Lol. It uses so much more. Read the pdf instead of skimming the initial paragraphs. Your home-brew algorithm with 27 tests is no match for the thousands of comparisons that the various functions in this package can run on your corpus.

And again: R is one of the most commonly used programming languages to perform statistical analyses on large bodies of data. The fact that you are unaware of its existence means that you do not know what you are talking about. You've been tinkering on your own, away from the community of digital humanists and stylometry people in particular. And you are unaware of basic tools used in the business. FREE tools.

"You have explained that "Iain Banks and Iain M Banks" are 2 pen-names of the same person, so checking the texts by this author against each-other is as rational as checking 2 texts by any established byline if the author happened to use a middle initial in one of them. There is no mystery to solve here unless you are saying you doubt these two Banks are the same person; this can be the case if somebody else wanted to benefit from Banks' status and used a similar byline to fool readers into interpreting the texts as being by the famous guy."

The question was if your algorithm would correctly identify Iain Banks and Iain M Banks as one author. As a test for your method. One pseudonym was used for litfic, the other for SF. How different are these two bodies of work from each other? We don't know, but you claim to have an algorithm that does author identification. You also claim your method works. If you test all but one of his litfic books, and all but one of his SF books, would your algorithm correctly predict that those final two books were written by the correct pseudonym?

"There is no mystery to solve here"
No, indeed. We know these two bodies of work belong to one author. This is to test the reliability of your algorithm. If your algorithm can be applied to "mysteries", you have to show first that it works on non-mysteries.

"To find how many underlying authorial-styles are behind Ulysses, I would have to test a few of Joyce's other texts and dozens of texts of other authors published in James' circles/ by the same publishers/ in the same genres."

False.
You could chunk ulysses in several thousand-word bits and run your tests on those. See how likely it is that they are by the same author. Then compare the other Joyce texts using the same chunking size. Or compare the various chapters of Ulysses and see how many of those cluster together as written by "the same author." That would give you an idea of how many different "author signatures" are at work here. Or not.

If your algorithm only works to find other authors from your corpus in a given mysterious text (let's call it X), then of course that is what your result is going to be. If your algorithm explicitly tries to identify contemporaries of the author of text X as the ghost-writers, and then of course you are going to find them.

Authorship attribution algorithms are designed to correctly attribute Text X to potential authors. But they can also provide an author's profile, given a body of work that is attributable to author X.

"To find how many underlying authorial-styles are behind Ulysses (...) The tests would only indicate which segments of Ulysses were by different writers if there were around 100 texts in the corpus that included all of the potential ghostwriters with multiple samples from each. "

This is the crux of your problem. This, right here. Your method assumes that a shockingly low degree of similarity between texts (based on a mere 27 characteristics, mind you) is evidence of the author of one contributing to both. Your method finds ghost-writers because that is all it is looking for. Your method does not create author profiles based on one author's body of work, as you yourself admit. Your method only works if you compare two or more authors with the explicit purpose of assigning one text to another. It treats genre similarities as authorial signatures. It looks for similarities between texts, and ghost-writing is the only explanation it allows for.

Your methodology is broken. Genuinely broken.

>136 faktorovich: My confusion here is not that you aren't using R in your work - I agree, I'm also too lazy to use R when I play with stats! - but that you weren't familiar with what it was or the fact that it is (100%, totally, completely) free (for everyone) (with no hidden charges).

Anyone who has done even minimal study of statistics should be aware of R even if they choose not to use it. Since it's completely free, it's the most commonly used software for statistics, and it's used in most introductions to statistics - I was introduced to it in a beginner course on stats for non-math people.

It's very hard for me to understand how someone could have been as involved as you are with statistical work without immediately recognizing what R is when it was mentioned! Even if you've never had an opportunity to take a modern statistic class, if you'd made any attempt to review other people's work in the field you would have seen it constantly mentioned, too. Like I said - it's like somebody doing research on Rome who doesn't know what Latin is. I don't expect you to speak it, but if you don't even know that it's a language, it's hard for me to trust that you actually know anything about Rome.

>139 Petroglyph: Oh I have definitely decrypted it! LIke I said it's a map of Oak Island, where the secret original manuscripts of Shakespeare that were written by Roger Bacon are hidden. Elizebeth Friedman solved it first but the FBI stopped her from sharing her solution, it's all encoded in the folk song "The Schooner I'm Alone."

(I am absolutely definitely not taking it seriously. If I was I'd be using R.)

>110 LolaWalser:

I can answer some of your questions.

"How is "style" defined for the purposes of CL (computational linguistics) analysis?"

For identifying an author's "style", you take some of that author's undisputed works (at the least several tens of thousands of words, though methods exist that can leverage smaller quantities of text), and then apply various tests. Here are a few common ones: the n number of most frequent content words (nouns, verbs, adjectives, adverbs); The n number of most frequent function words (pronouns, prepositions); Parts of Speech (i.e. how many nouns, verbs, adjectives, adverbs, prepositions, pronouns, etc); average word length; average sentence length; and vocabulary richness (the proportion of different words to total words -- some authors use a more limited vocabulary range than others). You can use strings as well: the n number of most frequent 5-character strings, or 3-word strings (or more).

A combination of all this can be used as a shorthand for identifying an author's style. Not in the sense of deep-reading their texts, but in the sense of how this particular author uses the language they write in. It's a mechanistic approach that looks at features that are hard to fake. Especially the sequences of three characters (or five, or whatever) -- there's no real way of faking that. People just don't pay attention to that sort of thing when producing meaningful texts and arranging information in accessible and/or pleasing patterns.

Note that I separated content words from functional words earlier. Functional words (such as pronouns and prepositions) are often not used in identifying a style, because they pattern differently across genres (a simple example: "I" and "you" are much less frequent in non-fiction texts, so any differences in their distribution across a novel and a tractate cannot be taken as evidence of difference of authors). Obviously, if you're dealing with the same genre, they're fine to include, and they're quite telling.

In order to run these tests multiple times on the same work, texts are often divided into chuncks of x number of words (500, 1k, 5k, 10k are common; accuracy tends to increase with chunk size).

Software packages exist that offer out-of-the-box stylometry tests: once you get your texts in a machine-readable form, and once they have been run through a tagger for Part of Speech, you can point the stylometry algorithm at your corpus and it will automatically perform the segmentation into 5000-word blocks (or whatever size you've specified) and apply a battery of measures to them.

These algorithms are continually refined and adapted: other features are thought up and tried out, and different algorithms apply better or worse to certain types of texts.

"--is CL analysis accompanied by other, traditional forms of analysis, such as, er, comprehensively reading the text?"

Well, these tests are often run on corpora of many millions of words. It's impractical to deep read all of these. Computational methods were designed to deal with quantities of texts that humans could not possibly digest, and to extract patterns that humans could detect only painstakingly and with ridiculous investments of time and effort.

Analysts use qualitative judgements when applying algorithms to texts: identifying one author across different genres requires a thought-out selection of features looked at (or rather: excluding certain features). If two texts compared feature a male vs a female main character, then the distribution of the pronouns "he" and "she" cannot be taken as an argument to confirm or deny authorship, for obvious reasons.

Results have to be interpreted, too, obviously, and this is where qualitative thinking is paramount. The numbers are just that: a measure of similarity along multiple parameters. What they mean depends on the case at hand.

These are some of the qualitative features taken into account in addition to the quantitative analysis.

"One of other confusing points is the assumption about shared style--that it's all or nothing, or that style doesn't change over time or may not change deliberately etc. (it's a whole host of questions really)."

Yup. Writers often change style as they age. I wouldn't be surprised if, say, Austen's later texts were different from her earlier ones.

"I can't" and "I did not"

Yeah. For one thing, Faktorovich limits herself to the six most common three-word strings in the text. That's basically useless. Proper authorship and stylometry studies routinely look at many hundreds of the most common word/phrase/string. I'm not exaggerating. Hundreds.

For another, those three-word phrases are red herrings. By her own admission. She says in >119 faktorovich::

"The 3-word-phrases (the top 6 most-frequently-appearing of these out of all possible 3-word phrases in a given text) I pointed out as revealing in the Brontes' case are not one of the 27-quantitative tests involved in the basic method. I collected these 3-word-phrases for all texts and used them to find obvious patterns of phrases that only appear in the work of any given authorial-style. The use of a contracted vs uncontracted "I can't" is a significant stylistic divergence that is revealing when one of these appears among the top-6 most-common phrases because this means there are many instances of this preferred usage in the text vs. the alternatives. I did not use any 3-word-phrase patterns to establish attributions, but just used them in the writeups to show that the less easily understood mathematic matches vs. non-matches were also confirmed by these verbally-descriptive elements."

So she's looking at the top 6 most frequent three-word strings. No word on if the three-word string in question is number 7 the other Brontë's text, or the tenth. No word on absolute frequencies, either -- is the difference four times versus two? (i.e. basically irrelevant) Forty versus two? (Potentially interesting) And in reality, she's not even looking at these phrases at all! She arrived at her authorship attribution using a paltry 27 criteria. And in order to make the mathy numbers more understandable to her audience of non-specialists, she looked at the top six three-word phrases and used those as shorthand, deciding that they were equally diagnostic as her more mathy results. Again -- no word on absolute or relative frequencies. Just the fact that they are among the top six in one author and not in another (though perhaps in the top seven? top ten?) is treated as corroboration of her "mathy" tests.

It's not top six that is important, it's differences in proportions. And she admits that she hasn't actually looked at the actual differences in proportions for these strings -- only top six vs not top six. It's ridiculous!

Were books edited in the Brontës' time the way they are now? Because things like contractions can be very dependent on editorial decisions, I could even see a typesetter working off a handwritten manuscript making particular decisions about contractions.

>142 melannen:
Sounds intriguing. Hit me up if you want me to apply some inscrutable methodology to your data that'll confuzzle the masses in their un-eliteness.

>122 amanda4242: Oh my heavens, you ghost-wrote all those authors novels? I'm a huge fan!

>144 Aquila: I was thinking that, too. Some publishers have certain words, phrases, spellings that they insist on/don't allow as part of a 'house style'. Whether the printed work uses cannot, can not, or can't is probably a lot more dependant on publisher and typesetter than on the preferences of the author. It's also not something an author is likely to get up tight about. There are plenty of other issues for that.

>147 MarthaJeanne: I was thinking the methodology is more likely to be identifying scriveners or clerks making fair copies for the printer's use from the original manuscript complete with spelling errors, amendments, tiny text (to fit more words on the page), interpolations and annotations and so on. In other words, it's the publisher taking the author's working copy and turning it into the printer's working copy.

It would then make sense that it would look like 6 ghostwriters on the grounds that this would be consistent employment for a clerk, who could be working on multiple manuscripts throughout a year.

>144 Aquila: I also have some big questions about how standardization of spelling and punctuation, or rather lack thereof, is being taken into account in the analyses.

I work as a copyeditor, and my experience is that writers -- even writers working in this modern age of computer word processors -- aren't particularly consistent. It seems to me that some elements of her analysis (questions of word choice, etc.), would need to recognize that "traveller" and "traveler" or "&c" and "etc." are the same words, whereas other ones (identifying writerly idiosyncracies) might need to treat them as different.

This is particularly relevant since she is working with texts that were written at a time when the written language was treated as rather more fluid than we do today, and as a result the spelling and punctuation choices of 17th-century authors are likely to seem somewhat arbitrary to contemporary readers. Anyone who has read a non-modernized version of Shakespeare or Chaucer will have noticed this.

On top of this there are issues of textual history and editorial preparation of texts, which may further obscure the "signatures" that the analysis depends on. In an era in which manuscripts were handwritten and either copied by hand or manually typeset, a lot of variation creeps in, some of which is not author intention but may be dictated by mundane reasons like the typesetter using an abbreviation or a shorter spelling to cram in a word at the end of a line.

When these texts are published today, they therefore undergo editing in one form or another. Depending on the intended readership, this may involve making changes to unify or lightly modernize spelling and punctuation, or it may involve comparing different versions of the text (e.g. First Folio/Second Folio) as part of the creation of a scholarly edition including a commentary that lists variants in the textual tradition and indicates missing/illegible words and similar matters.

It isn't at all clear to me what version of these British Renaissance texts are being used, except that they are versions freely available online. This doesn't tell us anything about the editorial preparation of these texts.

>149 spiphany:
If you look at her github repo, you'll find her bibliography here. (click on "view raw" to download a .docx file). I see a lot of Project Gutenberg and Early English Books Online.

>150 Petroglyph: My point was more that she doesn't mention editions at all when discussing her methods, and this hardly seems irrelevant to me given the era in which these texts were written. "Freely available online via Project Gutenberg" isn't a particularly compelling criterium for selecting which edition to use.

>70 paradoxosalpha: "I'm just sitting here with a well-buttered barrel of popcorn" - that's the funniest thing I have read in this thread so far!

>106 susanbooks:: re: " OMG when will this end" ... well said! The lady is certainly intent on defending her thesis! I suspect it will end when the buttered popcorn runs out...

>149 spiphany:
Ok, I got curious how stylometry papers on Elizabethan plays handle these things and I just looked at this paper: Eisen, Mark, Alejandro Ribeiro, Santiago Segarra, and Gabriel Egan. 2018. ‘Stylometric Analysis of Early Modern Period English Plays’. Digital Scholarship in the Humanities 33 (3): 500–528. https://doi.org/10.1093/llc/fqx059.

(if you encounter a paywall at that link, there's a pdf link here).

Segarra et al. say this on p. 3: "When using original transcriptions we have to account for the fact that many words had multiple accepted spellings during the Early Modern era. E.g., the word ‘of’ is also spelled as ‘off’, ‘offe’, or ‘o’ whereas the word ‘with’ may also appear as ‘wid’, ‘wyth’, ‘wytt’, ‘wi’, ‘wt’, and ‘wth’. Many of these alternate spellings are used infrequently and thus do not contribute highly to the WAN of a text. Nevertheless, we correct the WANs so that the occurrence of any of the alternative spellings is treated as the same word. We emphasize that spelling preferences carry little information about the authorship of a play. Indeed, spellings in printed editions were not necessarily those of authors as they were often selected by printers to accommodate the fixed length of lines in printing presses [33]."

The reference behind [33] is this: Philip Gaskell, A new introduction to bibliography, Clarendon Press Oxford, 1972.

In other words, these spellings are unified -- there's lists of common alternate spellings of function words, and an algorithm replaces the alternatives with the "standard" one. Because they're more the printer's area than the author's, as you pointed out.

I may have missed it somewhere but all the texts being discussed are the ones which are identified as misattributed.

What texts had been used to prove that the algorithm actually can recognize proper attribution? Surely, anyone proposing an algorithm has done extensive testing with both positive and negative use cases.

So how about sharing 5 pairs/groups of texts from the same time (as they would have undergone the same level of editing and so on during printing) which pass this test on attribution? I cannot imagine that a properly run research has not gone through a lot more than 5 sets of these.

>6 Crypto-Willobie: Down here in Australia I believe our equivalent colloquialism is "bullshit". Certainly by the intensity of the thread it's "baffling brains". The lady certainly is intent on defending a position that would appear to be under heavy fire! I note she even ripped into Mr Spalding's light hearted response. Perhaps with all the references in this thread to the Bronte sisters we may conclude the good Doctor has a "bee in her bonnet".

>9 DuncanHill: ... it's always a good day when I learn a new word! "Nephelokokkygia" ... I thank you, sir!

>79 anglemark: I think Erich Von Daniken must be due for rehabilitation as well!

Perhaps the good Doctor's theory could be best summed up as "I reject your reality and substitute it with my own".

>102 faktorovich: " it is illogical to claim that any writer could have made a living as a "professional" writer by only publishing as few as a single book or staging a single play in a lifetime.

But many people publish one or two books without ever intending to make a living as a writer. They may have spent years writing their one book idea as a hobby. Or thought they might like to be a writer until faced with the realities of how much work is involved after the writing and how little profit there is in it. And even those who still want to be professional writers find that they still need a bread job. This is true now, and I think it was probably true then. If you look through LT, you will find lots of authors only credited with a single book.

I would like to add, too, that the stylometry package me and melannen talk about (e.g. in >134 melannen:) comes with a sample of full-length novels by different authors, for learning and testing purposes (before users learn to prepare and import their own datasets). From the notes:

This dataset contains a selection of 9 novels in English, written by Jane Austen ("Emma", "Pride and Prejudice", "Sense and Sensibility"), Anne Bronte ("Agnes Grey", "The Tenant of Wildfell Hall"), Charlotte Bronte ("Jane Eyre", "The Professor", "Villette"), and Emily Bronte ("Wuthering Heights")

Somehow the multiplicity of authorial voices in the Brontë novels has gone completely unnoticed in a standard set of test novels that's been used by nearly every user of this package. Curiouser and curiouser!

>145 Petroglyph: I will keep it in mind! I have a few thousand more pages of defensive wall o' text to write first I think.

>149 spiphany: et al.: Actually, similar stylometry has been used to try to distinguish between editors/compilers/typesetters and things like which actors dictated some of the worse Shakespeare folios. With handwritten manuscripts (like the Voynich, seriously this time) it's also been used to even distinguish between copyists or different scribal centers.

It would be interesting if this methodology was actually detecting something like different Project Gutenberg copyeditors or Elizabethan print shops' punctuation standards (and hey, typesetters and copyeditors get even less credit than ghostwriters!) But given the description of the methods I suspect any signal it does have is completely drowned in noise and other signals that can't be separated out.

>137 Petroglyph: My method combines 27 different tests. Each of these tests have different measuring systems. For example some are for the number of punctuation marks in 100 sentences, while others are for the percentage of passive voice. These numbers cannot be compared to each other without changing them into the same types of measurements. The point of the experiment is to compare all texts in the corpus to each other in similarity. So, the goal can be reached by simplifying the data into similar and dissimilar data-points. While in this step the data is simplified, then the 27 tests are combined, creating 27X the precision of merely counting any two texts as being within 17-18% of each other or outside this range. The 17-18% change is needed because in a spreadsheet I select the same number of rows above and below a given data point to choose the similar or proximate texts, so there might be 2 above and 2 below or 5 above and 5 below, whereas precisely 17 or 18% might have meant having to choose between 4 below and 5 above or 5 below and 4 above; with equal number below and above, the percentage they add up ranges between 17-18%. I have found this range to be statistically significant because it is over the 1/5 odds that are pretty difficult to beat in a random roll of the dice. It also simply works best in identifying similar texts when I have applied it to practical experiments. While it might seem that linguistic density is more significant than the top-6 characters, when I have tested this assertion, I have found that either of these tests can either be the best measures of similarity for a given couple of texts, or can have a glitch where they show similarity, but most of the other tests show a divergence. My method works because the combination of these different tests minimizes the impact of even a few glitches on any of these tests from incorrectly changing the attribution.

>138 Petroglyph: You have avoided answering my question by instead insulting my intelligence. "Statistics" just means the analysis of data. My 27-tests method thus uses statistics as it analyzes data. I simply disclose the exact steps and tools I use in this analysis that makes it replicable by anybody in the public and your methods do not. No, stats are not too hard for the average person. There are online free tools that make even extremely complex statistics easy as you just plug in the numbers and the software applies the relevant formulas. Thus, a method is not uniquely sophisticating if it asks users to use inaccessible software, ​instead of the software that can solve the same problems with other steps and is available for free.

The explanation you give after accusing me of distrust is nonsensical when one defines the terms you are using. Here's what you're saying: X methods use linear equations (2-dimensions between 2 points) to simplify a corpus tested for frequent-words into 2-3-dimensional two-dimensional (X-Y) plot diagram. You are thus incorrectly defining the steps involved in PCA, which basically standardize the data from multiple dimensions into a simple 2-dimensional plot can allows for all types of comparison to be visualized together. My method performs a similar standardization of the data from 27 dimensions or 27 different test-types into a single table (which can be visualized in a graph as well) that compares each of the texts against each of the other texts on their degree of similarity on the 27 tests put together. In contrast most previously attempted computational-linguistic methods just test for word frequency - one of my tests; they make this simplicity seem complicating by calling each word they test for an n-gram or a dimension of the process.

There are 7.8 million words in the corpus of 284 texts that I measured from the Renaissance. I tested all of these 7.8 million words, so I could say that there were 7.8 million dimensions or features, but also could multiply this number by 26 as I performed 27 different tests and only one of them was for n-grams (words), and one for n-grams (letters) etc. Why would words frighten me, are you forgetting that 12 volumes of my series are translations of previously untranslated Early Modern English into Modern English with extensive annotations that explain linguistic choices etc. made in these texts?

>140 Petroglyph: You are repeating yourself without reading my responses. I stated previously that there is no logical reason for anybody to use a computer language to create new software to test linguistics when these tools for counting punctuation etc. have already been built. When I have reviewed R and other tools I have concluded that they only do simple tests for words and letters and the language describing these steps just make them sound like thousands of comparisons by counting each word in the texts, or each unique word in the text or the like. Can you list a hundred or even a thousand of these tests this tool can perform. Why are you referring to these tests abstractly. What thousand tests can possibly be performed on a corpus; there aren't enough different linguistic categories that are relevant to hit even a hundred tests; there are only so many different punctuation marks, word-types etc.

I also explained before that I refrain from testing post-1926 works because they have not been digitized yet and are not available in full-text in the public domain; this is why I can't test your proposed M. initials mystery. And as I said in the rest of my reply that I ignored, it is possible there is indeed a mystery behind the M. initial, as another author could have used this distinction to profit off a popular name, so the name similarity alone is not enough to establish certainty in the authenticity of any given set of bylines.

The accuracy of any computational-linguistic method increases as the word-size of a given sample increases. Any samples under 1,000 words are like to produce too many glitches. The size of Ulysses is also enormous in terms of word-count, whereas the other Joyce texts are small, so this corpus would be skewed towards having overwhelmingly more Ulysses and this is likely to create additional glitches in terms of similarity.

My method does not choose which texts are included in the corpus. The researcher makes this selection. You are saying that if a researcher deliberately searches for contemporaries in the same circles with the tested author X; the researcher will find a ghostwriter in this mix. This assumes that most popular/research-worthy tests are ghostwritten. If there was no prevalence for ghostwriting; then, testing authors in a shared circle would not show several bylines sharing a linguistic signature.

I am willing to test Ulysses against all digitized Joyce texts in full and in pieces and against around 30 other texts from Joyce's circle to determine attribution patterns; if you are going to agree to perform tests of your own on the same corpus of whole and pieces texts and we can post the results on GitHub, and link these results to this discussion. Just email me so we can confirm the texts we are using and the chapter-breaks etc. we will be applying.

No, my method does not consider "shocking low similarity" to be proof of similarity. Just the opposite is true. I have no idea why you are saying nonsense and claiming I said it. My method does create a linguistic "profile" when it groups texts with a shared linguistic signature together and establishes the elements that the texts in that group tend to share. Your approach is flawed because you believe all of the bylines, even if texts within these bylines do not actually match each other or register as similar when quantitative comparisons are applied to them. If you believe all bylines; you have errors that are engrained in your method that lead to a re-affirmation of established bylines instead of to the true bylines. Your method makes biased assumptions, whereas mine is entirely logical and unbiased as it ignores all bylines until after the math establishes linguistic similarity and divergence.

>141 melannen: Advanced statistics was a required course in my undergraduate economics degree. I received an A. I have a filmographic memory, and I am sure we did not learn anything about any programming languages in that course. We did all statistics by-hand or with a calculator. I have no idea what statistic courses would cover programming of any sort. Statistical programs are written by programmers, so that statisticians can just use these programs without each of them inventing a program of their own to perform the same basic statistical steps. Programming is irrelevant to measuring linguistic features because the tools for these measures have already been programmed and are available for free. I hope you will stop repeating the same nonsensical criticism.

>143 Petroglyph: The opposite of your statement regarding the significance of the top-6-most-frequent words is true. In contrast, it is your approach of counting the "many hundreds of... words/phrases" that is "useless". The data for the top-500 most-common words becomes impossible to fully evaluate in an unbiased way because most of the same words should appear in all of the texts, while there will also be as much as half or more words that only appear once in one of the texts. Methods that use this approach never explain what logic is used to establish similarity when this many words are compared. Are all single-word instances discarded as irrelevant? Are the top most-common words discarded as well because they are too common: this is a common element mentioned in research of this type, as researchers are proud to have taken the most common words out of this comparison. In contrast, my 6-words method derives a couple dozen common top-6 words (and letters) patterns. For example, both "Charlotte" and "Emily Bronte" between their three tested novels share the a-pattern in words, which includes these six most-frequent words: the, I, and, to, a, of. These patterns reveal the personality-based preferences of the author that do not change significantly even in different genres.

Your criticism of my 3-word-phrase analysis is again nonsensical. The top-6 most-frequent 3-word phrases are obviously 1-6th in frequency, not 7th or 9th. I ignore all other 3-word phrases for this verifying test so that my analysis is standardized and I am not occasionally adding bias to my method by considering as you said 7th or 9th phrases in similarity to the most common ones. The size of the text determines the relevant absolute frequencies, so it would be irrational to compare these absolute frequencies in a corpus of 284 texts with texts that vary between 500 words and over 300,000 words.

>144 Aquila: The Brontes' novels were published by the same publisher, and thus the same typesetter/editor worked on them, so their editorial style/changes would have standardized all texts to include or exclude contractions, so none of these texts should stand out from the others if this was something the editor had standardized.

>147 MarthaJeanne: It is extremely difficult for an editor to adjust the percentages of punctuation and word choice enough to change the signature to match the editor instead of the author. But I have seen it occur in the "Shakespeare" corpus where modernizing/translating editors have changed the signature from "Shakespeare's" original style to their own. If this rate of editing was needed for the "Brontes", this would imply extreme rates of misspelling and grammatical errors similar to a change of the language from Early Modern English into Modern English. If we assume the author or the ghostwriter did most of their own work without extreme intervention from an editor, editorial input does not alter the attribution results of my 27-tests.

>148 Maddz: There is absolutely no relationship between who transcribes or copies a manuscript and the linguistics the original author used in the text that is being automatically copied. There is an enormous quantity of spelling errors in the final published texts from the Renaissance, so if there were any clerks, they were not editing these texts for spelling errors at least not on a scale you imagine. I found one curious text where the transcribers/ typesetter(s) appear to have inserted spelling mistakes deliberately in a set of six or so different printings of a single book in single-copy runs. You really should read my series to understand why your argument is false. The 6 ghostwriters were the writers, and not typesetters who edited the texts they were typesetting so much they obtained their own authorial style.

>149 spiphany: Most of the texts I tested from the Renaissance were taken from the Early English Books Online database. I switched to preferring this platform after I realized that modernized-spelling versions did not match the original-spelling versions because modernizing editors had altered the linguistic style. Based on my tests, when most of the texts in the corpus are using old-spelling, the glitches of alternative-spelling words are in similar proportions in all texts, and thus this becomes an insignificant abnormality that does not prevent the recognizable modern-spelling words from being classified by the counting software. While the Renaissance exclamation/question rates were different from modern authors; they show patterns among themselves that are easily recognizable as these authors unique authorial styles.

And it is unclear if all manuscripts were first handwritten, as some plays might have been written directly during typesetting; if all plays had been first handwritten, there should have been more manuscripts of the original handwritten versions that should have survived from this period.

Yes, I am aware of scholarly editions that are made to explain spelling variations. As you appear to have failed to notice, my Re-Attribution series includes 12 volumes (so far) with these scholarly translations with annotations that explain variation in spelling, invented words, typos, differences between editions and the like.

The full bibliography of the texts I used and their sources is available on GitHub as well as in Volumes 1-2 of the series. I describe the process I used for cleaning up the texts to delete all content by modern editors, and glitches created by digitization etc. in the methodology chapter of Volumes 1-2. I used a systematic method that allowed the texts to be compared as close to their original-form as possible.

>150 Petroglyph: Thanks for pointing this out.

>151 spiphany: You are referring to a test on Twain, Dickens etc. I ran in around an hour based on a request made by somebody in this chat. These Twain/Dickens texts do not have the rate of spelling variance that are present in Renaissance texts, so they have not been edited much since their original publications in the 19th century. It is thus perfectly logical to use their Project Gutenberg versions, whereas in the Renaissance corpus it is better to use EEBO. The editions I used are listed in the bibliography included in GitHub.

>154 Petroglyph: The explanation from Segarra is illogical. They point out that they have found varied meanings for the same words such as "of" and "off", but they suggest they standardize all spelling to one of these variants in all cases; this would introduce many more errors than it would fix in terms of the processing of word-types etc. by the software. I have found it is best to leave the original spelling alone, only editing glitches created by digitizing software etc. It also makes no sense that authors had no input in the spelling of the words in their texts, as this suggests authors were dictating these books orally to a printer who was choosing the spelling.

>155 AnnieMod: As I explained before, most of the texts with the bylines of the 6 ghostwriters matched each other in the Renaissance corpus. For example, all of Ben Jonson's three tested plays (Volpone, Sejanus and Every Man) matched each other as well as the rest of Jonson's linguistic group, or the texts Jonson ghostwrote under other names or anonymously. You can see this in the 284-texts "Publishers" table in GitHub. And all of Percy's self-attributed texts, the two plays ("Cuck-Queans" and "Fairy Pastoral") as well as the one sonnet collection he published all matched each other, as well as Percy's ghostwritten etc. projects. The four texts tested from Josuah Sylvester all matched each other linguistically, and a couple of them were collections that included several texts vs. just one text attributed to Sylvester. And both of the two rare texts Verstegan self-attributed to himself texts that were tested also matched each other as well as the rest of his group. As did 2 out of 3 Harvey texts, with the one exception being a letter exchange that Harvey had with Verstegan that primarily matched Verstegan's signature without Verstegan's name appearing in a byline because Verstegan had been exiled from England. Only 1 poetry collection of Byrd's was available for testing, so it could not be compared with other texts of Byrd's. Similar tests on most other bylines proved that multiple ghostwriters wrote the texts with a single shared byline. This rare in-byline consistence for the ghostwriters contributed a small part in proving the case for them being the underlying ghostwriters.

>168 faktorovich: "The Brontes' novels were published by the same publisher"

In fact, they were not. Emily's novel & Anne's first novel were published by Chapman & Co. Charlotte's novels were published by Smith & Elder. Later, Smith, w/Charlotte's encouragement, gathered all of the Bronte novels & published them but the originals were not at all pub'd by the same house. Any brief biography will tell you that.

It will also tell you that Charlotte & Anne went to London to present themselves to George Smith to prove they were indeed the writers of the novels he was publishing. He was satisfied. Manuscripts, letters, diaries all confirm the Bronte authorship. Further, research might have shown you that Branwelll Bronte was assumed by many to be the author of his sisters' novels bc no women could have written such racy books. No one believes this now. Except you, I guess -- but you aren't even proposing Branwell but A B Nichols, who was considered a bit of a dimwit (even by Charlotte herself, though she eventually married him). If he was a co-author of the texts, you'd think he'd have cared about the Brontes' literary estate, rather than stuffing everything in an attic to mold over. There are so many historical & biographical reasons you're wrong, but keep pushing those numbers -- at least it amuses you, if no one else

>166 faktorovich: I'm sorry your one undergraduate statistics course didn't go into the kind of tools that are used by people working with statistics in real life! Statistics teaching in the US is really bad - it's a problem because it leads to people misunderstanding what statistics can do and not realizing when they're misusing them. My sister just finished her PhD and basically had to teach herself how to use R for her research because it was standard in her field but she hadn't had any stats classes since undergrad either. But since she'd taken the trouble to learn about what other people researching the same thing were doing, she knew that R was something she needed to know about.

(R is a program as well as a programming language - you don't have to write programs from scratch to use it! And packages like Stylo exist precisely so statisticians don't have to reinvent the same methods from scratch over and over.)

>177 melannen: But she got an A in an undergrad class!

>175 faktorovich: The GitHub contains the list of the texts you used for this specific research on the alleged mis-attributions where you applied the tests/algorithm. What I am asking about is a few sets of texts which were used when the algorithm/tests were developed and tested. Surely they were tested on a much larger body of works with clearly defined control sets (both negative and positive) which prove that the algorithm does what it claims to do before applying it on texts whose attribution is alleged to be wrong and using it to determine authorship.

Or were the 27 tests built to look for specific things in the Renaissance writers' works based on what the developer of the tests expected to see and fine-tuned for these specific works? If so - how was decided which of the texts are correctly attributed and which are not so the control tests can be ran on them?

If the tests are not fine-tuned for the period, what happens if you grab 100-200 Victorian novels and run the process on them - does it differentiate the authors properly? How about 100-200 novels from the early 20th century? Or any other period which has enough available tests online. I cannot imagine a published study could have missed to do at least that much of a due diligence so that should be a very quick answer?

>178 susanbooks: In advanced statistics, even!

Shouldn't feeding time be over by now?

>160 MarthaJeanne: An author only writing a single book in a lifetime is equivalent to a whale only having one meal in a lifetime. To spend a decade writing only one book even while holding a full-time job means writing on average 27 words-per-day for a 100,000-word average-sized book. The previous sentence has 25 words in it. So, you're saying an author sits down, writes a sentence, and then does no writing for the rest of the day? Is this really a believable concept for any human that has a passion for writing?

>161 Petroglyph: This means you have an erroneous testing method that failed to distinguish similarities because it based its attribution on the bylines, instead of allowing the linguistic data determine the attributions.

>176 susanbooks: Again, I have written 2 pages about the Brontes, and you are focusing on it as if it is the center of my research. "Emily" is the byline that disappears in my analysis that merges it with "Charlotte's" style, whereas "Anne's" style is divergent from both; so if "Emily" and "Anne" were published by the same publisher, this means the objection that different publishers could have introduced elements that caused the divergence in their styles is incorrect. This was the point of my statement; I did not list the irrelevant for this point full publishing history of these texts. Your argument about Nichols being a "dimwit" is extremely biased and irrational. What proof to you have for his stupidity, or for his isolation in an attic, and if he was isolated in an attic, don't you see a similarity there between this biographical point and the mad woman in the attic in "Charlotte Bronte's" Jane Eyre? I already explained why the "Bronte" documents being sold by Sotheby's at this time were never available for handwriting analysis previously (instead being hidden in closed archives), and that they were likely to have been forged by the forger in whose hands they were first found.

>177 melannen: Now you are insulting my statistics professor? It was the most difficult statistics class available at UMass (R1 university); the teacher wrote the leading statistics textbook at the time; and I had to take calculus before I qualified to take that class, so it was not basic-statistics by any stretch, but rather equivalent to what I would have learned in graduate statistics classes.

If the R program was indeed easy to apply to perform linguistic tests on a corpus, you would be able to write a paragraph summarizing the steps we can all take to use this tool similar to the basic steps I provided for my method. You have not attempted to do so because these steps are deliberately withheld from the public to force literature scholars interested in attribution to hire computational-linguists to do it for them without having access to the steps to check if these computational-linguists made any errors in their analysis.

>178 susanbooks: I also had a 4.0/A GPA in my graduate PhD program. Are you equally excited about that achievement as well?

>176 susanbooks: "Emily's novel & Anne's first novel were published by Chapman & Co."

Wikipedia seems to say that they were published by Thomas Cautley Newby.

>179 AnnieMod: My 284-text corpus is the largest number of texts ever tested in any computational-linguistic study of the Renaissance to-date. And you are asking if I tested my method on other texts to check the validity? Sure, it so happens that I did previously tested the method on the 18th century British texts (this study is forthcoming after I finish the second half or another 14 volumes or so in the Re-Attribution series), as I mentioned earlier, and then on 19-20th century texts (this data is available on the GitHub page). You have failed to read my earlier replies before making these criticisms.

While the 27 tests were chosen that better fit the Renaissance vs. 18-century texts, these tests could not be manipulated with any type of bias as they were calculated with basic counting software. For the 18th century, measures such as words-per-sentence could be used because there were few plays in this corpus etc., whereas this measure was not a good fit for the frequent broken play-lines in the Renaissance. The corpus was expanded gradually, and at each expansion point, all newly added texts were compared to the earliest added texts among the rest; this was a constant re-testing that repeatedly confirmed earlier findings.

>187 cpg: Thanks for the correction. It is irrelevant who published them for the specific attribution test that I ran. My goal was not solving who the ghostwriters were between these "Bronte" texts, but rather simply to point out that the masculine bylines used in the original texts were more likely to be the accurate gender-assignment. To establish if their publisher/editor/relatives etc. were the underlying ghostwriters, all of these potential authors have to be tested linguistically and biographically. Just checking the name of the original publisher(s) only solves the issue I addressed that the linguistic signatures do not show a publisher-specific pattern based on the evidence you guys are presenting.

>182 faktorovich: And yet it happens. I've got a LibraryThing page for two pages in a book. People write one book and lack the impetus to write another. https://tellersofweirdtales.blogspot.com/2018/11/f-georgia-stroup-1882-1952.html tells of a woman who wrote one story (a great one, by my tastes) and one nonfiction article, and apparently nothing else that was ever published.

And this is why I found your claim about mathematical purity obliterating bias so frustrating. If you believe that every author is a full time author, then it's not surprising your results show that a handful of authors wrote a vast array of works.

>184 faktorovich: This, like several other posts in this thread, show a carelessness in reading other's texts. Susanbooks did not say anything about Nichols being isolated in an attic.

>102 faktorovich:

it is illogical to claim that any writer could have made a living as a "professional" writer by only publishing as few as a single book or staging a single play in a lifetime.

Harper Lee

>190 prosfilaes: I am not disputing that no author has ever only written one book or one article in a lifetime. I am defending against your false assumption that the large number of texts I am re-attributing to the Renaissance ghostwriters alone is sufficient to discredit my conclusion. While across the history of writing, some authors might have been unprofessional or non-full-time writers; as I mentioned earlier in this exchange, independent authorship was illegal in the Renaissance according to vagabond laws without an aristocratic patron. So the past claims that hundreds of Renaissance lower-class people managed to publish books on contract for payments without any affiliated aristocrats is the obviously false claim; and not my interpretation of this period as a time when six ghostwriters fraudulently bypassed the vagabond law by using multiple pseudonyms to write for a living.

>191 lilithcat: It is possible that if I tested your Harper Lee counter; I would find that the style in Mockingbird matches that in other bylines, or Lee might have indeed only had one book in her. Thus, naming an author known for only one book cannot possibly prove or disprove what the odds are that anybody who only wrote a single book is likely to have used a ghostwriter.

I recall well the enormous amount of work required to take the Project Gutenberg files for the first five Tom Swift books to prepare them for a special 100th anniversary edition in 2010. My wife and I spent several hours for each by making a close comparison with the published copies from our collection. There were an enormous number of changes in punctuation and other details that needed correction.

This is only a single example of why texts from the public domain era may reflect many hands introducing changes in spelling, punctuation, and even altering the words themselves. On many classic works of literature there are variorum editions that document the differences between early texts for a given work.

For this reason, I am a bit skeptical of counts of things like exclamation marks in a single copy of a text from an online source.

In this long thread I have seen references to 27 tests but no listed summary of those tests.

In my studies of Edward Stratemeyer, himself an employer of ghostwriters through his Stratemeyer Syndicate, with one of his principal publishers, Warren F. Gregory of Lothrop, Lee & Shepard of Boston. He complained about the editorial and typesetting changes made to his typed manuscripts. The publisher defended some of these.

Further I have seen publisher manuals from the early 20th Century which gave details on the style rules applied by that publisher when they would typeset a book. Some of this same information was formalized in "manuals of style," some of which are still followed today.

It is perhaps not thought of by all readers but should be recalled that there are many hands and minds between the "author" and the printed page seen by the reader. These can influence the final product but also introduce noise that makes it harder to get to the initial author and their pure output prior to self-, typesetter- or publisher-editing.

James

>194 Keeline:
From the interview: "punctuation, lexical density, parts of speech, passive voice, characters and syllables per word, psychological word-choice, and patterns of the top-6 words and letters"

Plug any long-ish text into one of the three online services she lists in step 2 in that interview. The tests originate from there.

>165 faktorovich: "Can you list a hundred or even a thousand of these tests this tool can perform. Why are you referring to these tests abstractly. What thousand tests can possibly be performed on a corpus; there aren't enough different linguistic categories that are relevant to hit even a hundred tests; there are only so many different punctuation marks, word-types etc."

Challenge accepted!

Leaving aside like four punctuation marks ("!", "?", ";", "("), and the POS (let's say 9: Noun, Verb, Adj, Adv, AuxV, Prep, Pron, Conj, Interj), and a few thousand word types. Leaving aside avg word length, avg sentence length, function words, content words, unique words, character n-grams and word n-grams and string n-grams (for multiple values of n), passive voice, emotional content, etc. Here goes:

• potentially diagnostic word endings (-ed, -ly, -n't, 'll, -ing, -ion, -age, -dom, 's, ...)
• potentially diagnostic abbreviations: etc, eg, ie, viz, ...
• Sentences starting with "The", sentences starting with "It", senteces starting with "There", ... (This is not the same as testing the distribution of these words across two texts/authors/genres. Specifically: is there a difference in the distribution of how these authors/texts/genres start their sentences?)
• Sentences ending with: prepositions in general; pronouns in general; on, of, to, by, ...; me, you, him, her, it, ...
• Sentences starting with conjunctions in general; sentences starting with "and", "if" (as opposed to authors who prefer the if-clause at the end of the sentence), "however", "though", "thus", ...
• Proportion of Nouns/Verbs, proportion of Ajd/Noun, proportion of AuxV/V, ...
• Combinations of Adj+Adj+Adj+Noun; Adj+Adj+N; Adj+N; AuxV+AuxV+Verb (e.g. "might have done; could have done; ...),
• Proportion of Nouns per sentence; Verbs per sentence; Adj per sentence; ...
• proportion of ",/."; proportion of ";/.", proportion of "!/.", proportion of "?/."
• Numbers: units, tens, hundreds, thousands, ...; fractions; dates; currency; ...
• clauses starting with "which", or with "what", or even with "that" (i.e. introducing relative clauses)

I'm sure there are more. Sentences starting/ending with common two-word combinations; sentences starting with common three-word combinations. Diagnostic phrases (One of the reasons they caught the Unabomber was because he said "you can't eat your cake and have it, too" instead of the more usual "have your cake and eat it"). Dialectical features (which can be morphological, lexical, syntactical).

There you go. Hundreds. Not just "counting words and punctuation". Several of these count specific targets in specific contexts (sentence-initial words; sentence-initial phrases; word-final morphemes); several count proportions of various POS and the proportion of, say, questions per declaratives. Just like your test for psychological word choice, counting Numbers is not just "counting words", it is counting a specific subset of words that are more typical of some text types than others. Just like your test for Passive Voice isn't just "counting verbs" (it counts a subset of AuxV+Verb patterns with a specific usage pattern and stylistic preferences), a measure of sentences with relativizers counts a set of clause types that are associated with the complexity of the text and with the depth in which sentences are embedded in each other (more relativizers inside relativizers generally means more complex texts).

The "thousands" would include all the (tens of) thousands word types (all the occurrences of I, you, me, is, are, work, works, worked, working, ...) in the text and check for unique/diagnostic expressions. Not top-six or whatever. Unique words. Again, which ones those are you cannot know a priori. They have to emerge from the data.

Some combination of these factors with uniquely identify a Jane Austen text from a Emily Brontë text, such that if you apply that signature to another of Austen's works it'll correctly identify it as by her with a confidence of over 95%. You could do the same for my comments in this thread and yours. If one of us posts a new comment, a good author signature will correctly identify the new post as belonging to either of us, and if your selection of features is fine-grained enough, you can get shockingly high levels of confidence (95% is kinda low). Some combination of these might distinguish Iain Banks from Iain M Banks (or perhaps the two are indistinguishable -- there is no way of knowing until we try!). Some combination would probably distinguish the individual chapters of Ulysses. Some combination would distinguish your fiction from your non-fiction.

Of course, the vast majority of these will be irrelevant, or rather: their relevance (if any) will be so tiny that they're negligible. But a couple won't! And -- this is the important bit -- you can't determine which ones a priori -- this is something you must determine empirically, by trying them out! By finding exactly which ones work for each author!

Your method uses 27 tests. That is all. They are 27 common ones, to be sure, and they'll get you part of the way. But they will not suffice to correctly and reliably separate hundreds of authors across multiple centuries and multiple genres and multiple social classes and multiple geographic areas and multiple ages and multiple styles and multiple levels of education etc. etc. They are not nearly fine-grained enough, by at least an order of magnitude. You need more fine-grained analyses to even begin tackling a task of this magnitude. And yet, you are limited to the ones that the free online services you use spit out. Your methodology is unable to run tests that the online generators doesn't hand it ready-made. You are unable to even check if "sentences beginning with "there" are a diagnostic difference between two authors, or if a certain proportion of "There" sentences coupled with a certain range of Passive Voice avoidance with a certain proportion of function/content words. All your method does is assess some similarity between texts based on only these 27.

Not even on a subset of these, either. No: always these 27 taken together. Whether they are relevant to those particular authors/genres/periods/... or not. Would your method even be able to reveal an author's unique/diagnostic words? Does any of the online services you use do that? Would you be able to if the online service doesn't? Can you run your own unique-word test on a particular text and see if it's different from the generator?

The truth is: You did not select these 27 tests because they were diagnostic, or relevant, or appropriate for 17thC texts, or appropriate for 1800s novels. You chose them because the online tools give them to you. And that is not enough to do the kind of far-reaching research and the kind of heavy lifting that you want them to do.

The "similarities" you find between texts are so wide and so non-fine-grained that they hide diagnostic differences for all of the hundreds of tests that you are unable to test for, and that you are, apparently, unaware of (why else ask me to list some?). The fact that you only use 27 tests for 284 texts between 1560 and 1650 is directly responsible for your method returning a limited number of individual-signature authors.

Incidentally, stylometry software allows you to implement any of the above tests, just like that. When you say in >129 faktorovich: that "The R programming language is not relevant to my analysis because the basic tools for counting the number of punctuation marks etc. have already been programmed and are available for the public to use" this merely reveals that you have no idea what you are talking about. In your interview, you say that "My combination of 27 different tests is thousands of times more accurate than the standard method in this field, which only tests the frequency of common words." Lady, you do not know what you are talking about.

Next up: genres.

The similarities that your method reveals between texts can be due to those texts belonging to a certain genre. But someone using your method has no way of knowing.

Some combination of these hundreds of features be able to differentiate between, say, comedies and tragedies (a bundle of features that are more "chatty" and informal, for instance); or fiction and non-fiction (the latter will have a relative lack of contractions, and more passive voice, and a greater diversity of conjunctions, and a greater lexical diversity, and a number of others). Non-fiction texts will also contain more numbers, and a greater diversity of numbers (units, tens, hundreds, fractions, years, currency, ...) and more function words.

Your method tests for some of these (but not all), and it, apparently, cannot tease bundles of features apart. There is no way of uniquely and reliably identifying a text as non-fiction with your method. A similarity on, say, function words or passive voice (each of which is worth a 1 in your method) is likely to be cancelled out by, say, another feature with a "0". There is no convenient way of making certain features weigh more or less than others.

You've applied the exact same 27 tests to comedies, tragedies, historical plays, letters (I believe), and novels. There is no way that the results for all of these can be correct to equivalent levels.

Now, you may perhaps say that you don't look at non-fiction texts. But that's not the point. This is just an example of a genre that ought to be easily detectible. Your method, if it works, ought to be able to distinguish works that are similar-because-same-genre from similar-because-same-author. I'm sure that Shakespeare's tragedies share similarities with Marlowe's. But your reliance on a) only 27 tests and b) always these 27 tests prevent you from distinguishing genre similarities from author similarities.

Proper stylometry software is a couple of steps above free online passive voice counters. They can handle this sort of stuff. Your methodology can't.

>165 faktorovich: The size of Ulysses is also enormous in terms of word-count, whereas the other Joyce texts are small, so this corpus would be skewed towards having overwhelmingly more Ulysses and this is likely to create additional glitches in terms of similarity.

Again, you seem to not know what you are talking about. Anyone who looks at authorship attribution and who uses things like average sentence length and lexical diversity has to take into account the differences in the lengths of texts. This is very very obvious. So obvious that stylometric analyses routinely employ a standard solution: chunking. (search for "chunk" on this page, and you'll find I've mentioned this technique before).

Texts of different lengths that are compared are routinely sliced into 500-word chunks. Or 1000 words, or 10,000 words. Or whatever is appropriate. You can have each text chunked hundreds of times -- multiple stretches of text will be contained in many chunks, obviously, but one reason this is done to prevent over-fitting: fitting your averages too close to a particular set of chunks (say, near the beginning of the novel, which has different features from the ending). In this way, you can compare many texts of different lengths on things like average sentence length and lexical density, because the averages are calculated on identically-sized chunks.

This is such a bog-standard methodology in stylometry that it's not even explained anymore beyond a mention of which chunk size was chosen.

Applying chunking in stylometry software is a breeze: specify the chunk size, and the number of chunks to be chosen, and you're good. Your methodology, which involves a lot of manual copy/pasting from .txt to webpage to spreadsheet, with manual sorting and manually changing percentages to 1 and 0, would not be able to chunk easily. It'd take so much time and so many clicks and changes of environment and cut/paste. Again, your comment that "The R programming language is not relevant to my analysis because the basic tools for counting the number of punctuation marks etc. have already been programmed and are available for the public to use" makes it seem as though you are completely unaware of the kinds of things this thing can do.

Do you do visualizations? R does...

>188 faktorovich: Questions are not criticisms.

"these tests could not be manipulated with any type of bias as they were calculated with basic counting software"
The software does not decide what to count or what data to use from the data points that got counted, the software user does. And that's where the bias comes from.

>194 Keeline: Your reflections on the nature of editing and publishing are irrelevant to the questions at-hand. You are discussing 20th century publishing rules, when the series that we are discussing in this thread is about the British Renaissance. The degree of editing changed enormously over the centuries. As you should know there was a wide array of spelling variations for words that was acceptable in the Renaissance, which would all be corrected as errors by modern publishers. What you consider to be an "enormous" number of changes might or might be statistically significant for a computational-linguistic analysis. Were there more than 1 change per 100 sentences; less than 1 change would not even alter the percentage of given words/ punctuation marks per 100 sentences. I have read original and edited versions of Dickens and Twain and the first editions are very similar to the latest edited editions; these are the authors I tested in the mini-test made upon request, so you have to look at their texts in Project Gutenberg if you want to find fault with their over-editing. I named the 27-tests in the interview and repeated them in this thread. They are for: punctuation, lexical density, parts of speech, passive voice, characters and syllables per word, psychological word-choice, and patterns of the top-6 words and letters. 3 of the 6 Renaissance ghostwriters were publishers themselves (Byrd, Verstegan and Jonson), so it was their job to edit/typeset these texts. You guys really just need to read my book before making these statements, as if you had read it you would understand why these points are irrelevant and false when they are applied to the British Renaissance.

>129 faktorovich: Sure, I can apply the tests to myself. You should know that I have done professional ghostwriting in the past, though I cannot disclose for whom. You would have to test millions of texts with around a million bylines to figure out what I have ghostwritten, as it is only a few projects in a sea of modern publishing. It would be less extreme for me to just test the texts with my own byline, but only testing my own tests against themselves would once again fail to establish what they are different from. For every text by me, I would have to add a few texts by other bylines writing in the present (and again I try to avoid testing anything published after 1926). Testing works by any single given author produces a set of numbers that represent the range of that author's style; only when this style is compared with other styles do these tests indicate authorial attributions with my method.

>165 faktorovich: I am willing to test Ulysses against all digitized Joyce texts in full and in pieces and against around 30 other texts from Joyce's circle to determine attribution patterns

Look. We've been here before (my final comment in 140). Your methodology turns up ghost-writers almost everywhere because you are not allowing your method to do anything else. If your method is indeed applicable to, like, all of the public-domain authors from the 1560s through 1926, then it should be able to tell whether one of Joyce's works is different from the others, and just how far removed it is from them, and which of his other public-domain work is closest to it. It should be able to reveal which of your own texts cluster together, whether your fiction texts are systematically different from your nonfiction texts, and whether your interviews cluster differently from your monographs. No need to involve other authors or texts.

Instead, you only use it to compare texts with your non-fine-grained 27 tests with the specific intention of attributing similarities in text signatures to the same individual author. When asked to compare your works against each other, or Joyce's works, or Iain (M) Banks' works, you automatically make the jump to involving other authors and texts specifically to see which texts are similar-ish, specifically so that you can, in your own words, "determine attribution patterns". Detecting ghost-writers is not the only thing your methodology should be able to do. Assessing an individual author's signature across their body of work should be no objection.

Assuming the conclusion, garbage in garbage out, not testing for alternative explanations. Yadda yadda.

Look. I know you worked very long and very hard on your method. And it can't be easy to hear that you've made some basic errors that invalidate your entire methodology and all the work you've built on it. If you choose to entrench in your sense of being right, I can't stop you. But I hope that I've highlighted some of the very fundamental issues with your methodology. As it stands, I can trust exactly none of the results of this methodology, because you're trying to make a basic set of unvaried tests do way, way more work than is plausible. I hope you take this on board and strive to improve your methodology. Some of the basics are sound, but some of its shortcomings are fundamental errors that cripple the whole enterprise.

You're enthusiastic. You have the drive, the time and the energy. The kind of analyses that you are trying to do cannot be reliably done with the kind of free online text analyzers that you use. You need more advanced tools. Perhaps a programming language, perhaps Statistics with Excel.

I know three things:

It is not my responsibility to peer review your work (even though I might as well have, lol).
It is not my responsibility or my job to explain again why your methods are ureliable and produce false results.
If these explanations aren't sufficient, nothing will be.

Peace. Here ends my peer review. I'm sure there will be much rejoicing.

>196 Petroglyph: I just wanted to say how much I've enjoyed your contributions to this thread. I've always thought linguistics a fascinating field, and your posts have taught me a lot...even though some of it's over my head!

>202 Petroglyph: Thank you, Petroglyph.

>185 faktorovich: I'm not insulting your statistics professor! I'm complaining about the state of statistics teaching in the US. I too took my school's most advanced undergrad stats course, taught by one of the great experts in the field - you had to have passed linear algebra and multivariable calculus. I learned absolutely nothing that was useful for actually doing the stats work in my science and humanities classes, which is why I ended up also taking the stats for non-math-people class, which was much more useful.

We're on the same side here about the academy! I also agree that there are a lot of things that are gatekept and obfuscated in academia, which stifles new voices and helps bad ideas propagate. But you're barking up the wrong tree here - R is well-known partly because it has done a lot to make stats *more* accessible to non-specialists and people outside the Academy. The first example where R was brought up in this thread was somebody doing a stylometric analysis for fun, as a hobby! That's why I feel so strongly about this - it's really important that programs like R mean that good, rigorous stats are something anybody can easily do at home.

Here's how to run a stylometric analysis using R in less than fifteen minutes:
1. Go to Project Gutenberg. Download txt/UTF-8 files of some texts you want to analyze. Save them to a folder called "corpus". I downloaded a bunch of Bobbsey Twins, since we know they were ghostwritten and people like keeline have done the documentary research to figure out who the ghostwriters were.
2. Go here: https://www.rstudio.com/products/rstudio/download/#download and dowload and install the free desktop version of R studio. (That website also sells pay versions because it's a slicker version of the free software, but you don't need the pay version.)
3. Open your newly installed program. Go to "tools"->"install packages". Type "stylo" and press "install".
4. In the "console" tab, type "library(stylo)" and press enter.
5. Then type "stylo()" and press enter.
6. This brings up a new window (it has a quill pen icon) with a bunch of options. You can adjust the options, but the defaults are pretty good. The only one I changed was I went to "output" and selected "jpg" so that my results will be a pretty graphic.
7. Press "okay".
8. Navigate to the folder your "corpus" folder is in and press "Select folder".
9. The window will go away and you'll see some stuff flash in the "console" tab. Then, go to the folder you selected and you'll see a .jpg file. Open it. There is a tree diagram showing how similar your texts are to each other! There are also some other files that have the word tables that Stylo used to make the tree, so you can load them into a spreadsheet and do other things with them if you want.

There you go, that took me less than half an hour including installing the program, finding the corpora on Gutenberg, googling a tutorial on how to run Stylo (since I'd never used it before and haven't used R in years) and then writing these instructions.

That isn't going to get you rigorous results, of course - for that, you have to do the work to understand the statistics that Stylo is running, and how to select your corpora correctly, and what settings to use for various things - but if you want an easily reproducible-by-anyone method, there you go!

(Other people in this thread: you should play with this, it's fun. I am peacing out of this discussion here too because I have now installed R and am distracted playing with Stylo instead.)

>196 Petroglyph: You did not list a hundred and certainly not a thousand different tests. What few thousand word-types; there are 4 word-classes (individual pronouns cannot be counted as a word-type in this context as this is statistically nonsensical - and it is better to just use the most-common-words)? The biggest problem with most of the tests you did list is that the Renaissance texts include irregular punctuation (sometimes no periods at the end of sentences), and strange spellings/endings (that would skew your -ed/-n't endings, among other measures). And some of the meanings of words are different or the spelling is so different your machines would not recognize the intended word to classify it as an adverb vs. verb. Commas and semicolons can also be mistaken or mixed up randomly, so measuring their order might confuse your system. And there are no commas before "which" that are common in Modern English and you might use for this metric. If there is no period at the end of a sentence, your system would not be able to count what type of a word begins the next sentence. Thus, while I considered using some of these more complex measures, I decided the basic tests resulted in the smallest number of glitches and the most consistently accurate results. And while you can list this large list of possible tests, I have read over a hundred computational-linguistic studies and none of them described any such list, but instead tend to just say they test for word n-grams but in language that makes this sound convoluted. Can you cite a study with a list like the one you gave and with raw-data publicly available of the matches/non-matches/ quantities of all of these measures that resulted using the R program?

If you select "unique/diagnostic expressions" on a text-by-text basis you are using your biased judgement to measure only the words you cherry-pick as relevant without any stated system for what this selection is based on in all cases.

I know that a combination of linguistic tests correctly identifies the linguistic signature groups of tested texts: this is precisely what my method is. I simply use the right tests in a method that reaches consistent correct results because it is unbiased, whereas your method is designed to re-assert existing bylines even if these are incorrectly grouping texts by multiple signatures.

Your concept of "diagnostic words" is faulty because it assumes that anything other than the most-frequent words can be compared systematically in an entirely unbiased manner. I previously explained how your method's omission of the most and least frequent words and the use of unexplained special "diagnostic" words means you have given yourself a license to pick out which words are tested in a text without explaining how you have excluded the words you are not considering. Thus, your results can be entirely what you desire them to be and not what the data might have said if your goal was the truth and not hitting an over 95% accuracy rate based on currently-accepted bylines.

There are far more than 27 tests available on the platforms I linked to, and I did indeed choose these tests because they are relevant to the Renaissance corpus, as I have repeatedly explained.

There has not been any previous study that I have come across that has used 27 or more different types of tests, as I have done. Just send a link to this mystical study you are imagining that includes this list of hundreds of tests and more than my corpus of 284 texts from the Renaissance, and let's digest it together to see how it varies from my approach.

>203 amanda4242: >204 Aquila:
Thanks!

>205 melannen:
Very well done! I really wish I had your brevity.

>197 Petroglyph: Yet again you have not actually opened the spreadsheets in GitHub to see the data I provide in them. If you open the Publishers spreadsheet, you will see that I specify the genre for all of the tested texts in a separate column. Some of the 6 Renaissance ghostwriters, like Verstegan, worked in several genres including non-fiction, drama, translation, novels, etc. Others specialized in poetry (Byrd), and a couple worked in both poetry and drama (Percy/Jonson). Within the dramatic genre, Percy preferred writing tragedies, and Jonson preferred comedies, but they also wrote in the other genre in a portion of their work. All this data indicates that my method identified the underlying author even if he wrote in multiple genres. The genre did not change the attribution calculation. An author who prefers a high quantity of commas, will keep adding a lot of commas in a drama as well as in non-fiction. Any small genre-related glitches were not significant enough to impact the combined 27-tests attributions.

>199 AnnieMod: The software counts the same type of measurements in my tests; the user does not change the tests for different texts; thus, my method does not have bias, unlike rival approaches.

>201 Petroglyph: If you tried teaching college English for a few years, as I have, you would notice that plagiarism is extremely prevalent. I have taught in Georgia, Pennsylvania, Arizona, Ohio, China, Texas - just all over the map, and TurnItIn has detected up to 98% of papers being plagiarized in my students' work. The College Admissions Scandal is not showing an exception, but the cultural norm. A more relevant case is the revelation that "Carmen Mola" was not a woman, but really three male ghostwriters who won an award recently in Spain. I run a publishing company (Anaphora Literary Press), and have been working with writers and meeting writers at conventions etc. since 2009. I know why I am seeing ghostwriters, but I do not know why you are insisting ghostwriters aren't real.

Yes, I could test only Ulysses or only Banks or only my own texts, as I explained before; but these tests would not prove anything important. Testing individual works in isolation by selecting one or a couple of "Marlowe" texts and a couple of "Shakespeare" texts is how computational-linguists have derived the current nonsensical re-attributions of the Renaissance. Small numbers of texts in a corpus will lead to either matches or non-matches, but not to who the underlying authors are, or if a ghostwriter etc. was involved. Proving that a "Marlowe" is similar to one "Shakespeare" text has been sufficient in earlier studies to re-attribute one of these texts, but this is extremely sloppy and careless. Have you tested your own, Joyce's Banks' works in isolation; if not, why are you telling me I have to run this test my method works. I have already verified my method on several corpuses, which I guess you have not realized since you have not actually read my research.

>209 faktorovich:

I will be direct: you are an idiot

"LibraryThing prohibits personal attacks, name-calling . . ."

>202 Petroglyph: You did not perform a peer-review of my research, but rather repeated the talking-points that commonly come up and I have heard before in peer-reviews. This is why I have already thought through the points you raise, having previously sent similar responses without receiving clarification from "reviewers" like yourself who aim to exclude competing methods as their primary objectives for serving in the reviewer capacity. Your goal is selling your services as a superior computer-programmer, and making rival-researchers feel like they have to surrender their testing methods to you to perform the tests for them. You would then return the "answers" or the attributions that re-enforce established bylines, or are entirely nonsensical and seem to show no logical attribution is possible in a corpus. I have had this exchange with a computer-linguist before. I rejected these re-attributions because they were blatantly erroneous, and the researcher refused to provide raw-data, so I could check what went wrong. As I said before several times, no programming is needed in computational-linguistic author-attribution because the tools have already been programmed. You just profit from the opposite because you make a living from re-programming.

>210 faktorovich: The bias comes from determining what you want counted, not from using different counters. A person made a decision what metrics need to be counted. That introduced bias into the whole method - intentionally or not.

>206 faktorovich:
You did not list a hundred and certainly not a thousand different tests
Yes I did. The "..." in that list is crucial and does much of the work.
Your not agreeing with some of them does not mean that they can't be tested for. And how do you know they are not relevant for your texts if you reject them out of hand? That's assuming the conclusion, lady. It's a no-no.

The biggest problem with most of the tests you did list is that the Renaissance texts include irregular punctuation"
If your corpus has widely inconsistent punctuation... then don't test for it?? You do know that it is possible to not test for certain things, right? You don't have to!

"If you select "unique/diagnostic expressions" on a text-by-text basis you are using your biased judgement to measure only the words you cherry-pick as relevant without any stated system for what this selection is based on in all cases. "

False. The exact same kind of software that counts the "most frequent word" can count the words that occur more in a certain text per 1000 words (or whatever) than in other texts, relatively speaking. Not in absolute terms, in relative terms.

"diagnostic words"
"the words you cherry-pick as relevant without any stated system for what this selection is based on in all cases"

I ain't been picking no cherries, lady. "diagnostic words" does not refer to hapax legomena, words that occur only once in a text and which you can select according to your own biases. It means "words that author X uses more often relative to other authors. Look, the word "bias" is not very common in everyday speech, but some people use more than others. If you divide the total occurrence of "bias" by the total number of people who use it, you get an average. Some people use the word "bias" a lot more than others, and some people don't use it at all. Comparing an individual's use of the word "bias" to that average gives you an idea of how much that person's use of this word stands out from the average. If they use it more than the average to a statistically significant degree, then that word can be used as a diagnostic in helping to identify that particular person from the others. Those are the diagnostic words I speak of.

Some authors have these diagnostic "words" -- favourite shibboleths, typical expressions, a fondness for semicolons, a habit of including footnotes, a preference of "&" over "and", ... The list goes on. These things become part of their individual style -- tics they have that they do more than the average. And averages are easy to count across your entire corpus of text.

"Your concept of "diagnostic words" is faulty because it assumes anything other than the most-frequent words can be compared systematically in an entirely unbiased manner."

... you mean, like Passive voice, which isn't a most-frequent word, but rather the proportion of a particular pairing of AuxV+Verb pairs divided by all Verb phrases? Of course you compare more than just absolute counts. Relative counts is where it's at!

"There are far more than 27 tests available on the platforms I linked to, and I did indeed choose these tests because they are relevant to the Renaissance corpus, as I have repeatedly explained. "

By your own admission: these tests are less applicable to your 1800s corpus. Some of your results may be caused by an improperly-balanced set of tests.

"Can you cite a study with a list like the one you gave and with raw-data publicly available of the matches/non-matches/ quantities of all of these measures that resulted using the R program?"

Ramnial, Hoshiladevi, Shireen Panchoo, and Sameerchand Pudaruth. 2016. ‘Authorship Attribution Using Stylometry and Machine Learning Techniques’. In Intelligent Systems Technologies and Applications, edited by Stefano Berretti, Sabu M. Thampi, and Praveen Ranjan Srivastava, 384:113–25. Advances in Intelligent Systems and Computing. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-23036-8_10.

If you're hit by a paywall, here's a pdf

Not sure if they used R. That doesn't matter, really. They used a set of 446 features, derived from the literature (i.e. commonly used tests in similar papers and books) and from machine learning, and compared them to a subset of 135, finding that the 446 features performed better. See sections 3 and 4.3. If you want an exhaustive list of all the features they used, there are three email addresses on the first page. Researchers love to talk about their stuff to people.

This paper should be of interest to you: it's about identifying plagiarized portions of Author X in texts handed in by Author Y.

>205 melannen: I am going to check this method tomorrow. From looking over your directions, it is clear that the basic auto method only compares texts on word-frequencies, or on only 1 test, and the system appears to be selecting the words it chooses to test for the user. Thus the resulting data is likely to be extremely difficult to double-check to see how these words were chosen. Obviously the tables could not compare all of the words in all of the texts against each other as this would include billions or trillions or more comparisons; so the word-selection process must be biased. I doubt the system would even let me run this basic test for free, but I will write a review after trying it tomorrow. Why exactly do you imagine this method of letting a more difficult to setup program (than the simple-to-operate/open software I use) compare words (1 test) is more precise than my method of applying 27 different tests?

>213 faktorovich:
"the talking-points that commonly come up and I have heard before in peer-reviews.
The fact that you ignore and/or dismiss peer-review comments does not surprise me.

This is why I have already thought through the points you raise, having previously sent similar responses without receiving clarification from "reviewers" like yourself
Peer review does not work that way, lol. You send paper to journal, journal asks people who know about the subject to read it thoroughly and make sure that it's up to snuff.

Responses are usually

• "publish as is" (coveted, rare, oh shiny city on the hill)
• "publish with minor revisions" ("look, scholar, here's some issues that we had. Fix those and you're golden")
• "publish with major revisions" ("Some of this is a bit of alright, really. Some of it stinks. Don't publish unless massively rewritten")
• "do not publish" ("It's not up to snuff. Now go away")

You don't correspond with the peer reviewers. They don't know who you are, you don't know who they are. It's a double-blind kind of thing. If the journals you submit to don't do that double-blind thing, I've got bad news for you.

The journal, if it is worth its salt, has a solid tradition of selecting appropriate and skillful peer reviewers. If the quality of the published content goes down, people will notice and flock to other journals. There's an interest in keeping standards high.

Your goal is selling your services as a superior computer-programmer, and making rival-researchers feel like they have to surrender their testing methods to you to perform the tests for them

Crikey, enough with the ad hominem!

"As I said before several times, no programming is needed in computational-linguistic author-attribution because the tools have already been programmed"

Look. The sad truth is that Science Marches On. The easy things get done first. Then the less easy things. Then the hard things. Occasionally new areas open up where new groves of low-hanging fruit become available (e.g. ancient dna sequencing. Super interesting!).

But sadly, much of the low-hanging fruit has been picked. If you want to make new discoveries, you have to put in the work and understand how all the low-hanging fruit got picked, and all the medium-hanging fruit, before you really can start reaching for high-level fruit. Once you're there, the ladder you used to reach the low-hanging fruit won't be sufficient any more. You need better tools.

Also, some of the low-hanging fruit is rotten inside. You don't know until you've picked it and discovered worms or wasp eggs inside and have an explanation for why it is rotten. Some of the high-hanging fruit is rotten, too, btw.

"You just profit from the opposite because you make a living from re-programming"

>216 faktorovich: That program lets you do anything it can do for free!

The basic options in the window that comes up let you choose which and how many tests to run, and how many words to compare against each other, and how to select those words, and a lot of other things. It has tooltips explaining what each of the options does. You can also compare things other than words. There is also an option for putting in your own manual word or character list for it to use instead of generating one.

(You *can* have it compare all the words in all the texts, but it has a built-in warning that if you want to compare more than 5000 different words, you need a supercomputer. Still, that's dozens of millions of comparisons, and my cheap laptop did it in a few seconds just fine. Computers are good at doing billions of simple computations really fast! That's why being able to use things like R is so powerful.)

Those are just the basic options available in the easy-to-use window interface in my instructions - but there are also lots of other things you can do with Stylo and R, and plenty of free online resource explaining what they are. Figuring out what tests you should run is the hard part; once you've done that, it's easy in R to show other people how to do the same thing.

I am making no judgements about the precision of your method. I'm merely pointing out that it is actually just as easy to use the standard methods of stylometry, using free software like R that has the standard methods built in, as it is to use your method. You keep saying it isn't, and that one of the reasons your method is superior is that anyone can reproduce it. But it's clear you haven't actually done any looking into how to actually use the standard methods before you do that, because they are just as easy to reproduce! I hope you do try the stylo package and find it useful for your work. If you're presenting a revolutionary new way of doing things, it's really important to be able to demonstrate that you understand the standard way of doing things well enough to criticize it accurately.

Oh, more software that is absolutely free and will vastly improve your corpus-handling skills without requiring coding skills: Antconc. There were times in my life when I used that application daily. Good times. Miss ya lots, anty; gros bisous.

Oh wow: there's more freeware apps available there now than when I used it. That team's been busy like, uh, bees.

faktorovich, you might find antgram useful (for generating n-grams). Looks like you might have use for antpconc too: this tool analyzes parallel corpora. Lots of screenshots, and every utility comes with its own help pdf.

ProtAnt compares a UTF-8 text you feed it to a reference corpus you also can set. It'll show you the ways that the target text deviates from the reference corpus.

There's a file converter there, too, to create UTF-8 txt files from other formats.

Petroglyph, melannen, I just wanted to thank both of you for your patient, rational, and knowledgeable contributions to this discussion.

I realize this thread probably feels like a bit of a pile-on to the author, so I want to note that I do think stylometrics -- when applied to specific questions in specific contexts by people who know what they are doing and have taken all relevant factors into account -- has a lot of potential for interesting insights.

It's a pity the field hasn't necessarily been presented in its best light here, particularly for people who didn't have much or any previous familiarity with it. (Pennebaker's The Secret Life of Pronouns was my introduction to the topic; I thought he brought up a lot of intriguing points but occasionally reached too far in making sweeping claims about the explanatory power of the relative frequency of function words. But without this previous exposure I'm not sure I would have had any idea based on the blog post or this thread what the author was doing or why counting words and punctuation marks could tell us anything useful...)

>188 faktorovich:

And this is where it is obvious (if it hasn't been already) that you do not understand what AnnieMod and others are saying.

If we pick a set of books where the authorship isn't in question - does you methodology find differences or not. This is the same test people talked about with Iain Banks/Iain M. Banks or James Joyce. Arthur Conan-Doyle would also be a good test case.

If we pick a set of books where the ghosts are known (Stratemeyer) does you methodology identify them all as distinct individuals (and this should be inside a single series and across series)?

The idea is that you prove your methods work against a data-set where the results are known and undisputable, before you try them on a 'potential' unknown.

Members are reminded that the Terms of Service ( https://www.librarything.com/privacy ) prohibit personal attacks. Comment on the content, the argument, or the wording, not the person. Thank you for your understanding and for following this rule more closely in the future.

>184 faktorovich: "Your argument about Nichols being a "dimwit" is extremely biased and irrational. What proof to you have for his stupidity, or for his isolation in an attic, and if he was isolated in an attic, don't you see a similarity there between this biographical point and the mad woman in the attic in "Charlotte Bronte's" Jane Eyre?"

See, here's why some people aren't buying what you're selling. Every biographer since Gaskell (who knew him) said A B Nichols was a dimwit. Charlotte thought so, too, until she married him. Further, I never said Nichols was in the attic. I said, and again, you'd know this if you studied Charlotte only casually, Nichols put all of Charlotte's literary remains (including the pillar portrait) in an attic where they grew moldy & some disintegrated. (The fact that you'd take Nichols' storage of Charlotte's things in an attic after her death as proof of his connection to Jane Eyre is a sorry exhibition of your thinking)

As for Emily & Anne's writing being similar: they had been writing partners since their early childhood (again, any biography, however brief, will tell you this). We have manuscripts of their work, letters, and diaries, including from people outside the family (Ellen Nussey, for instance) that confirm this. Unless those master forgers wasted time faking juvenilia, much of which wasn't found until after all of their deaths.

If you'd ever actually read some of the authors you're talking about, you'd know that Anne is an ironic realist -- her tone is unmistakable and utterly unlike her sisters' who were more heavily influenced by the Romantics.

You don't know enough about your subjects to be making these grand statements. What is your PhD in? Certainly not the Humanities.

>187 cpg: Sorry, you're right, it was Newby. My bad.

>168 faktorovich: They were not, until later editions, published by the same publisher.

>221 andyl:

Actually on my Bobbsey Twins (+ a few Rover Boys and Uncle Wiggily) tests last night, it was consistently sorting the first book (which is supposed to be by Stratemeyer himself) in the middle of the books that are supposed to be by Howard R. Garis, while pulling out the ones by Lillian Garis accurately. Which is suggestive!

But given that I don't know what I'm doing and the set of texts was selected based on when I got sick of downloading them, not much more than interesting - I would need to actually learn what the different Stylo tests are good for and do some real tests with other texts, not just play around trying everything, before I was confident even starting to criticize the rigorous documentary research. (Playing around is the first step in science, not the final step.)

>182 faktorovich: "So, you're saying an author sits down, writes a sentence, and then does no writing for the rest of the day? Is this really a believable concept for any human that has a passion for writing?"

Woolf covered this in A Room of One's Own and Tillie Olsen updated that for the later 20th century in Silences. Women's Studies courses cover this, too. Again, your lack of knowledge about literary history makes your arguments absurd.

>203 amanda4242: ">196 Petroglyph: Petroglyph: I just wanted to say how much I've enjoyed your contributions to this thread."

Agreed, and the same to everyone who has been posting rationally.

I think only three people have been posting to this thread and I have the statistics to prove it.

>229 anglemark: I always operate under the assumption in any internet thread that it's just one person talking to themself. Especially if I'm on the thread.

>209 faktorovich:
No, you misunderstand.

Of course a single 500-word chunk is going to be unreliable. This is known. But where did I say that a single 500-word chunk is all that is taken?

But a 50,000 word novel can be divided into at least 100 500-word chunks without overlapping. Or 50 1000-word chunks. If you analyze all of these individually and then take averages across them (for instance), the quirks of individual chunks will disappear, and the repeated author-signature features will be amplified. You've gone from one block of usable text from which to take averages and frequency counts to 50!

There's methods that involve a movable chunk: rolling the window through the text. Say you take a chunk / window of 3000 words and analyze the frequencies and patterns etc for that stretch. Then you move your window 500 words ahead and start the next chunk at the 501st word. Redo the tests for that chunk. You continue rolling the window forward and testing each chunk until the window contains the final word in the text. That is your final window.

To state the obvious: many stretches of text will be sampled repeatedly: the last 2500 words in the first window are the same stretch as the first 2500 words in the second window. But moving a window of a few thousand words forward in steps of a couple of hundred words like that really reveals what odd/noteworthy/deviant frequencies of words, phrases, structures and grammatical or typographical phenomena are concentrated in a small portion of the text and, therefore, less relevant to the text/author as a whole. It allows you to distinguish one-off irregularities from repeating regular irregularities that occur in many windows across the entirety of the text.

This paper explains a few ways of doing this. This paper applies one of these methods to look at collaborative writing and to disentangle multiple authors in a single text. (If you want the pdf, ask and you shall receive.)

Instead of having one single measure of an author's language use (one text), you can now break up a single long text (such as a novel) into several dozen or even hundreds of chunks of useful size (usually 1k, or 3k or 5k -- depending on the nature of your corpus). It's a magnifying glass. Much more detailed look.

Doing this manually on an entire corpus is prohibitively time-consuming. The R stylo package has a function rolling.classify which does this automatically. I don't just mean sampling 3k words in steps of 500, but also doing the tests and keeping track of which phenomena are occasional irregularities and which are interesting. Other stylometry software has its own ways of implementing this technique.

This is how that Good Omens experiment was done, btw, with windows of 5000 words. (You need a test corpus of undisputed Pratchett texts and undisputed Gaiman texts first, obviously). When Gaiman suggested Callaway look at Pratchett's Pyramids, too, the method detected Gaiman's input in that book. Importantly, a book that the method had not been trained on. So the process works! It's an attractive method for someone looking to identify multiple author signatures (and where they occur) in a given novel. Of course, your choice of tests is crucial, too.

"I test texts in their entirety, as this eliminates looking separately at these parts that might have more glitches in isolation.

You cannot distinguish occasional / incidental "glitches" from author-specific irregularities and deviances from other texts in your corpus. Besides, this method tests the text in its entirety, too. Just at a much more detailed level of granularity.

You're thinking manually -- how to do these things yourself, by hand. That's just impractical! Computer-assisted methods put so much more power at your disposal. Computers are really good at doing this mechanistic and repetitive process automatically.

Methods that are this impractical to do manually in hand-crafted spreadsheets may seem esoteric and unachievable to you. But I repeat: Science Marches On. We're well past that stage now.

"In the articles I have seen that use chunking, they tend to select oddly different numbers of words per-chunk, instead of consistently using 500 or 10,000 etc. per-chunk"
I'm aware of chunking methods that have a fixed starting point (the 501st word, the 1001st word, etc) but that run to the end of the paragraph.

"this selective choosing makes it difficult to re-test a given text, and it means that some chunks are twice smaller than others, and thus have more glitches"
Really? One text's chunks were twice larger than the other? In the same study? Where did you see that?

By the way, it's not "selective choosing". It's a systematic, mechanistic process of working through the entire text in steps. There's no hidden bias here. Just maximizing the ways in which you can pull information out of a single text.

>228 susanbooks:
Thanks!

>219 Petroglyph: Oh, those all look really cool! Thanks for sharing the links. (The last time I messed with analyzing corpora it was for a stunt and I used Phylip, because I was trying to reconstruct the sequence a round-robin game was originally written in, but that was really not what that's supposed to be for!)

>229 anglemark:
My contributions to this thread have certainly been ghostwritten.

>230 melannen: I agree.

>205 melannen: As I expected, I downloaded RStudio, as you stated in the first step, but after downloading it, when I try to open it in my computer, I receive the message "RStudio requires an existing installation of R in order to work. Please select the version of R to use". So, I had to add the extra step of downloading R: https://cran.r-project.org/bin/windows/base/. Then, because most of the Renaissance texts are only available on EEBO, which only provides basic text content and not the "txt/UTF-8" format, I had to look up the steps for converting Word documents into UTF-8: https://support.3playmedia.com/hc/en-us/articles/227730088-Exporting-a-UTF-8-txt.... After saving a couple of files in the "Unicode (UTF-8)" format, I went back into Stylo Studio to check if these files could be seen by the program for evaluation, and found that the program did not recognize this file type as well as the simple plain-text I had tried before. Thus, I could reach step 8, but could not actually test this software to see if it works and what its results would be like. I have seen similar problems with all of the computational-linguistic programs I had tested in my research phase, and this is why I opted to create my own method and to use simple tools that are available without these complications/ glitches to the public. Do you know what the solution is to Stylo not recognizing the "UTF-8" files? If so, I will try to finish the steps to find out what happens. But this at the very least proves that this system is not for the general public, especially since even the basic steps you provided here are not available in any of the "Help" manuals I have checked in similar programs (I think I tried this program as well, and probably could not get far due to not knowing the special steps, and not having any instructions from the programmers.

>217 Petroglyph: "If the journals you submit to don't do that double-blind thing, I've got bad news for you."

In my field (mathematics), single-blind is the standard, not double-blind.

>215 Petroglyph: "Words that author X uses more often relative to other authors": This is why I use the most frequent words only, as this indeed establishes which words a given author uses more than any other words. Your methodology just turns the term "relative" on its head, by subtracting some words that are perceived to used less or more, or at the same degree out of consideration. My method just records the patterns of the most frequent words and compares them systematically without disqualifying or ignoring any of the words. "If they use it more than the average to a statistically significant degree, then that word can be used as a diagnostic in helping to identify that particular person from the others." Several biased interventions are required for the type of analysis you propose. 1. The choice of texts in your corpus can be manipulated so that "bias" is used with less or more frequency in the "average" for this group of selected texts. 2. How the researcher defines "statistical significance" is not likely to be stated in the article, and so a researcher has intervene to declare some variance from the average as significant while other variations are cherry-picked to be insignificant. 3. Your system invites the researcher to select a few "features" or words or punctuation marks that are chosen for comparison against the average, whereas my system evaluates all possible words and pre-determined tested punctuation marks simply on their frequency.

"Relative counts" becomes nonsensical in a corpus of 284 texts, as creating relativity and average calculations for every word is bound to create nonsensical results.

After some searching, I found the paragraph in the article you are citing that describes the "features" they tested for: "word length, character n-grams, function words, sentence length, punctuation, unique words, vocabulary richness, PoS tags and many more. There are also some new features like symbols, combined-words (e.g. web-based), word endings (‘ll, `ll, ed, ing, ion, ly, n't, s), sentences starting with the word “the” and sentences containing the following words (and, etc, e.g, what, which)." They claim to be comparing an experiment where they tested 446 features to 153 features, but this brief paragraph is all of the features they actually specify. In my experience, none of these computational-linguists respond to emails, unless they hope to be hired to conduct a study. If the point of their article was testing feature-quantity, they had to include the full list of features and to specify what the difference between the features was between the 446 and 153 tests. Their specified features are things that my method tests for as well (word-length, most-frequent characters, frequent words, sentence length, punctuation, linguistic density, mood measurement). The more rarified few features they mentioned are not applicable to the Renaissance where the language variance would create glitches if such tests were attempted. This article does not specify these features because they are counting individual words as features, or have other glitches in their chosen method. Since they say they are using their own software, readers cannot check their findings by running the data through their own software with the same tests. Incorrectly seeing no plagiarism where there is plagiarism with a biased method is far more problematic than simply ignoring the presence of plagiarism in PhD dissertations, so this paper is troubling vs. interesting from my perspective.

>217 Petroglyph: In one instance, I received an acceptance letter for a central article that explained my computational-linguistic method for the Renaissance into a periodical. However, the editor asked me to address the reviewer's concerns before a final decision could be reached. The "concerns" were nonsensical or made demands for me to insert irrelevant information, to alter my findings with falsified data, etc. The researcher asked me to do a couple of rounds of editing before insisting that I pay him to perform the experiment himself with his own method and to become a co-author on the paper. I refused to do so, and so the paper was immediately rejected, without an option for an alternative reviewer. If you think this is how scholarly peer-review is supposed to work in a society that hopes to make any progress in science/knowledge; you are in the same boat as this editor/peer-reviewer.

>219 Petroglyph: The concept of "generating n-grams" is nonsensical. An n-gram is any feature x that is being evaluated, like letter and word frequency. Basic counting systems calculate the most common words/letters etc. What is the point of generating letters/words? From what are you generating them, and to do what? My method analyzes any size of a textual corpus, including comparing linguistic signatures that can be visualized as being in their own sketched in corpora groups. So, now you are explaining that UTF-8 files can only be created with this special tool? What's the difference between the Word UTF-8 file vs the one this system creates? Can your Stylo program see one, but not the other? There are basically at least 5 different programs that have to be installed to use Stylo, and it doesn't even work with all of them - and you think this is a working scientific method that is applicable by literature specialists who are not programmers?

>220 spiphany: You can also read my Re-Attribution series where I explain my method and compare it to the previous computational-linguistic approaches in great detail that should be far easier to understand and far more informative than "Secrets".

>221 andyl: I have already tested my method in this discuss on 3 known authors (Twain, Dickens, and a third) and have shown that my method works. I have said that if any researcher in this discussion wants to put any larger set of texts to the full range of texts, I would be delighted to run the experiment with me testing the same corpus of texts as them (can be more than 1 researcher using more than 1 other alternative method). I proposed for all of us to post our raw data/ method on GitHub and to link to it in this discussion, and to discuss the results. Nobody has taken me up on this challenge. I have already made all of the tests I needed to on my end. So this experiment would be a test of rival methods, which is something I am curious to explain in detail given the lack of such raw data in published computational-linguistic articles as I have been explaining.

>222 timspalding: Thank you Tim for clarifying. Given the various types of insults that have been hurled at me across this discussion, I had assumed returning insults might have been a required part of scholarly dialogue in our modern age. It is good to find out this is not the case.

>223 susanbooks: If you were clearer in your statements about the relationship between Nichols and the attic, I could have addressed your intended meaning, instead of questioning if your attic allusion was referring to Jane Eyre. You are the one who is thinking about attics; it was not something that came into my mind in this context.

Yes, what I am saying that the evidence I briefly browsed in the Guardian's article about the Sotheby's sale is that a forger faked "juvenilia" after their deaths. This is not something I have spent any serious amount of time researching, this is just what the surface facts are blatantly saying.

I have closely read all of the Bronte novels I tested as part of my literature MA/PhD curriculum. "Ironic realist" vs. romanticism in tone: this is how you distinguish authorial style. I have written mystery, romance, fantasy etc. novels that have included romantic and ironically realistic tones; the genre/ tone an author uses does not define their quantifiable linguistic style. Tone is a conscious choice, whereas quantitative linguistic features are unconscious usage patterns that the writer cannot deliberately control. My PhD and first McFarland book are on 19th century British novels, or the "Brontes'" period.

>242 faktorovich:

I am sorry but the test you describe in post >124 faktorovich: does not address what I was aiming at. Post 124 seems to infer that for 1 text from Twin, Dickens and Bulwer-Lytton you can distinguish them as having separate authors. That isn't the point of my post at all. My point is can you take the entire corpus (or a reasonable subset) of Conan-Doyle and run it through your tests and does it give one unique author or does it not?

TBH as someone more maths minded I think any process with a binary output is flawed to begin with.

>243 faktorovich: No one, literally, no one threw insults at you. Their post would have been flagged just as yours was

>182 faktorovich: Have you ever heard of revising, rewriting, making changes ...?

>227 susanbooks: My MA thesis was about Woolf's Room, and my Author-Publishers book covered Woolf's publishing company. Nowhere in Room does Woolf say that women write only one sentence per day. Your assertion is false, and nonsensical. Room is about a woman having space and time of her own to write full-time as a professional, and this is the opposite of women being forced into Silences. You really have to use specific quotes from these books to support your arguments if you believe your statements have rational meaning behind them.

>238 faktorovich:
"Words that author X uses more often relative to other authors": This is why I use the most frequent words only, as this indeed establishes which words a given author uses more than any other words."

No, no, no, no. "Most frequent" is an absolute measure. Which phrase does this text happen to contain most often? In order for that to be meaningful, you'd have to compare the frequency of that phrase in your corpus in general, against other texts.

Most frequent does not mean "most diagnostic". It is sheer raw frequency. Unless you can demonstrate that a particular phrase occurs more than you expect it to, or more than in other texts, more than with other authors, sheer absolute frequency means nothing.

>236 faktorovich: I'm sorry I missed the download R step! I already had it installed so I didn't realize. It looks like the process of getting it via RStudio was pretty painless though - that's good!

Yes, completely free (no ads, no hidden fees, not web-based) software will often take a few extra steps, and they do have a little bit of a learning curve, but putting in a few days to learn the software is really worth it.

I'm not super familiar with EEBO. I used Gutenberg texts in my examples since I was familiar with that (and there are a lot of early English texts there as well.)

It's odd that a collection of early language corpora wouldn't have a UTF-8 download option! Most of the EEBO repositories I found with a quick google are only available to people with academic credentials, though (super frustrating!) so I can't really help you there.

It's super weird that it would only be available as .doc files though! Those are really bad to use for any sort of stats work, since they're full of irrelevant formatting and fancy encoding that can make it harder to get good data. Word files aren't intended for storing data, they're designed for making nicely formatted documents to print, so they don't work well if all you want is to store data. I'm not sure why it's not recognizing your converted .doc files. It might be something carried over from the .doc conversion.

It's actually a good thing that programs like this are often super picky about file format. File format issues can introduce a lot of bad data into your statistics, so anyone doing this kind of work needs to make sure they understand the way different text formats work and how that can affect the data they're getting. Especially with character-level analysis, being absolutely sure that, say, your file isn't getting confused about ` vs ' and that kind of thing is the sort of stuff that requires a basic understanding of encoding and text file formats.

Were you copy-pasting from a web-only HTML version of EEBO like https://quod.lib.umich.edu/e/eebogroup/ into Word? In that case the simplest thing would be to paste into a program like Notepad instead of Word - Notepad automatically saves as UTF-8 (UTF is just the simplest version of a .txt file, that's why it's so widely used.) You might lose some information that way, though. It would be better to find versions of those texts available that are already in a data-only format (maybe through one of the limited-access version of EEBO?)

faktorovich:

The method described in >205 melannen: works. There's a few things I want to add, though.

Rstudio is merely a GUI for R. You need both R -- the actual programming language that does the actual statistical analyses and generates the graphs -- and Rstudio -- which makes the programme look like software instead of a command-line interface, and which has areas for graphs and point-and-click things.

Don't use Word -- machines have a hard time reading that -- too many distortions from plain text. Plain text is where it's at. Test it on Gutenberg texts -- those come in plain text.

Same goes for tables. .xlsx files are proprietary, and mistakes may be made in converting MS Excel files into something more easily digestible by corpus software. .csv or .tsv are standard formats.

Stylo, by default, expects your corpus to be in a folder called corpus.

It expects that folder to be located in your "working directory", which is the default directory that R uses to look for files you tell it to read and where it puts the graphs and the other files it generates. If you're not sure which folder is your working directory, type getwd() ("get working directory"). This will tell you what folder R is in, and where it will look for a folder called "corpus". Put your corpus folder there.

If you follow the steps in >205 melannen:, R will create a graph for you (I'd suggest selecting a PCA under the "statistics" heading, with Classic delta as the distance). It will also dump into your working directory a word list of all the tokens in your corpus; a table_with_frequencies.txt, which gives the frequencies for all the word tokens in your corpus per text; and a stylo_config.txt file, which lists the configurations: the dimensions of your graph, which analysis was performed, the distance measure, and much else besides. If you want to recreate a graph / analysis: here are all the parameters you need.

Alright. Time for a Lunch Break Experiment.

I've used the following texts from ProjGut (selecting the ones without illustrations -- just plain text):
Austen: Emma, Mansfield Park, Northanger Abbey, Pride and Prejudice, Sense and Sensibility, Lady Susan.
A Bronte: Tenant, Agnes Grey
C Bronte: Jane Eyre, Professor, Shirley, Villette
E Bronte: Wuthering
Marie Corelli: Romance of two worlds, Sorrows of Satan, Young Diana

To state the obvious: I am aware of just how ridiculous it would be to compare these books for authorial attribution purposes (regardless of what you may think about the Brontes). This is a fun little exercise (well, fun for a nerd like me), a totally-not-serious playing around with some software with no pretentions to any solid conclusions!

I removed from these texts all of the Gutenberg legal text at the end, the Gutenberg text at the beginning, any prefaces, preambles, introductions, tables of contents. All texts start at Chapter 1. We're comparing fiction by these authors, not their introductions to various editions, or other people's prefaces. Properly speaking, I ought to also remove all the chapter headings from the text, and preferably any poetry etc. But for this Lunch Break Experiment, that's too much.

Austen: early 1800s, Brontes: mid-1800s, Corelli: romances from the late 1800s.

A zip of the Austen texts is available here. A zip of the entire corpus (Austen + others) is here.

In part one: How do Jane Austen's novels compare to each other? I expect that Lady Susan would be the outlier: it's the only epistolary novel (i.e. written in the I-form).

Here is how they cluster a cluster graph that I got by following the steps in >205 melannen:. I selected jpg under the Output tab, and Cluster Analysis under Statistics.



So. Northanger and Mansfield are fairly close; and so are Sense and Pride. These two pairs are more similar to each other than Emma. Lady Susan is hella different.

How does this look in a scatterplot? Again, using >205 melannen:, but with selecting PCA (corr) under Statistics.



Once you have a measure of how different these novels are from each other (or how similar), you can translate this difference into distances. You can then plot the distances between these novels on a single line, or on two, or on three. This graph uses two lines (the x-axis and the y-axis), each of which represents a different kind of difference / distance. It is up to the researcher to determine what these distances measure.

The closer two texts are in this graph, the more similar (or, properly, the more not-different) they are.

The percentages on the axis labels are how much difference is covered by them. The X-axis explains about half of the difference between these novels. The Y-axis almost a quarter.

The distances between the novels along dimension one, the x-axis, is mainly about separating Lady Susan from the others. This is clearly the only first-person novel that is massively different from the others (because of the difference in pronoun use). I can't really tell what the second dimension represents. Why are Sense and Emma furthest apart? Mans and North cluster together, and Pride is closer to them than to Sense. Off the top of my head I can't think of an explanation. But I haven't thought about this, at all. And I'm not going to spend more energy on this, either. The distortion caused by Lady Susan may be making other things less visible. If I wanted to, I could remove Lady Susan from consideration, and try again. But that would take me too far.

This is why you have to be careful with which pronouns or which function words your model tests for! This is why it is important to be able to remove certain distorting characteristics from consideration! This is why it is important to make sure that you are comparing apples with apples, or that you are comparing apples with oranges with the appropriate tests

Part two: let's plot the Austen novels, the Brontes' novels, and three of Corelli's novels together. Ridiculous, I know. Are they similar to each other? Different? Will Austen and Corelli stand out as separate authors? Will the Brontes? Why would that be the case?

Here is a cluster graph (same specifics as the Austen cluster graph):



Interesting!

• Austen's novels (in green) form a neat cluster (apart from Lady Susan, for reasons already discussed).
• Charlotte Bronte's books (in black) are more similar to each other than to other texts. Wuthering (in grey) is most similar to Charlotte's work.
• Corelli's books show up as their own cluster -- these texts are more similar to each other than they are to the others. There is a separation here between Romance and Sorrows on the one hand, and Diana on the other. The former two are third-person; the latter is first-person.
• Most interestingly, perhaps, is that Anne Bronte's work (in red) clusters with Lady Susan. The Anne books are closer together than to Susan, but still. And wouldn't you know it: both of Anne's texts are first-person narratives!
• This cluster graph thinks that these books fall into two main groups: Anne Bronte's texts, which are most similar to Jane Austen's; and Corelli's texts, which are most similar to the C and E Bronte.

So, the similarity between the Corelli books is much stronger than the separation third-person vs first-person -- her first-person Diana is not included with the other first-person cluster on the top of the graph. In other words: Corelli really has her own style that is very different from the others. (Spoiler alert: Corelli is famous for endless purple prose.)

Finally, I want to make this obvious: this graph does not prove, demonstrate, show, insinuate, hint at or imply that Austen wrote the Anne Bronte books. If pronouns were removed from consideration, these clusterings would look fairly different -- we've seen how much of the variability among the Austen novels is explained by first-person vs third-person. We'd have to run this thing again, without pronouns. But I leave that as an exercise for the reader.

Equally obvious, perhaps: There is almost a century between Austen's work and Corelli's. That is bound to have an impact.

Okay, on to the pca graph for these novels. Again, the only defaults I changed from the steps in >205 melannen: is that I chose PCA (corr) under statistics, and jpg under output.



So. The x-axis is mainly about separating the Austen books from the texts by Corelli and the Brontes. 35% of the diversity across these texts is covered by this separation. Is this because Austen's books are older than the others and the software is picking up differences between English-around-1800 and English-around-1850? Corelli deliberately affects archaic language, so that would explain why her books are closer to mid-century texts than you'd expect them to be. Perhaps there's an aspect of class, as well. Either way, I've gone over my allotted lunch time!

The Y-axis (which covers about 16% of the differences between these texts) is mainly about separating Lady Susan from the rest of Austen's work -- another confirmation that this text must be enormously different from the others. Corelli's texts fall below it; the Bronte texts above it, except for Shirley, which is a little closer to Corelli's texts.

Anne Bronte's texts (in red) are, again, closer to Austen's than the other Bronte texts, but not nearly as close to them as they are to her sisters.

Roughly, this graph puts Austen's texts on one side, groups the Bronte texts close-ish together, and finds a separation between Corelli's texts and the Brontes' (apart from Shirley).

Better grouping/clustering/distancing measures and a more fine-tuned analysis could perhaps turn this into a fun little blog post. But I've got to go back to work.

>119 faktorovich:

Thank you.

>143 Petroglyph:

And thank you, that's illuminating. I understand about the features being hard to fake but I think the real question is the one I note below, off a remark in >244 faktorovich::

Tone is a conscious choice, whereas quantitative linguistic features are unconscious usage patterns that the writer cannot deliberately control.

Not to be a smartass, but, Queneau, Exercices de style... handily disproves this. However, I understand why we'd assume most authors don't bother tinkering with such "invisible" parts of their style as interpunction, contractions etc. That's not (probably) much in question.

What is, as far as I can see is this--do those "quantifiable" aspects of style form absolutely unique patterns? From everything I've read here I'm tempted to conclude "absolutely not". When I think of applying this methodology to the type of writing prevalent in my field, I fear that the conclusion may well be the entire output is created by one, maybe tops two people. And there would be good reason to think so based on the neutral, boilerplate, standardised language we aim at--nonetheless, the conclusion would be completely wrong.

>237 cpg:
Interesting! Still blind, though. No bargaining with the referee!

>253 Petroglyph: Thank you for doing all this! I'm learning a lot.

>231 Petroglyph: No, the quirks of individual chunks do not disappear when you combine them with many other chunks; by combining or averaging them you simply create an output that is entirely nonsensical or does not represent the true authorial-signature in the text as a whole. In contrast, when you just test the text as a whole, you arrive at a precise attribution because you are testing the patterns that emerge when an author writes thousands of words (in this pool those minor glitches in the parts disappear). This is especially the case if a researcher selects chunks just as politicians in the US gerrymander congressional districts so that the final result is more elected officials that favor the gerrymandering-officials preferences vs. the average political preferences of the entire population.

Your point about rolling the window by re-testing some of the words many times while testing some of the words only once shows extreme bias and an extreme escape into statistical nonsense-land. The point of this "rolling" seems to be to create completely unrepeatable without the Stylo program experiments, as you say it is a method that cannot be duplicated with basic tools/math. By using a method that selects chunks in this unrepeatable manner, and in a format that cannot be shared in its raw form, researchers create an excuse for why they are not sharing the raw data, and thus they have the option to manipulate the data to give the conclusions that re-affirm current attributions, even if carrying out the test instead created nonsensical or entirely inconclusive results.

I explain the method for distinguishing collaborative writing in my Re-Attribution book. I reviewed all available scholarship on this topic as part of my research for this section.

...You are mansplaining how computers can be used to do some things but not others? That is pretty rude, and irrational given the depth I go into in my explanation of my computational-linguistic method and rival method in my Re-Attribution book that you still have not attempted to read. I am familiar with the available tools, and the computational tools I chose are the best ones to actually solve attribution mysteries. It is irrelevant how these basic working tools sound or seem to insiders in this field.

I have a filmographic memory, and I recall seeing a data set with differently sized chunks in several studies, but I do not recall the names of these studies. You can figure out which ones I must be thinking of by actually reading by Re-Attribution book, as I recall citing all of these articles in the annotations.

>245 andyl: Yes, the 284 texts in my Renaissance corpus, the 21 texts in my 19-20th century corpus, and the 100 or so texts in my 18th century corpus have already been tested to prove unique authors are correctly identified. As I have repeated, I am open to running a brand new extensive experiment on any accessible group of texts, if whoever is asking me to do it is familiar with the "standard" computational-linguistic methods and will run and share all of their data as well as me. I don't know why you guys keep challenging me without somebody stepping up to agree to accept the challenge on your end.

>247 MarthaJeanne: I spent a couple of years revising, rewriting and changing my study and the books in the Renaissance series before publishing these 14 volumes. Are you insinuating that despite being a literature scholar, I do not know the meaning of the term "revising"?

>252 Petroglyph: , >253 Petroglyph: This is all so lovely and fascinating! I'm sorry it's clearly falling on deaf ears (though the rest of the audience is rapt.) At this point you would certainly deserve a co-authorship if any article got through peer review.

No, I was confused that you seemed to think that someone would only go through the text of a book once during the writing of it.

>254 LolaWalser:
"What is, as far as I can see is this--do those "quantifiable" aspects of style form absolutely unique patterns? From everything I've read here I'm tempted to conclude "absolutely not". When I think of applying this methodology to the type of writing prevalent in my field, I fear that the conclusion may well be the entire output is created by one, maybe tops two people. And there would be good reason to think so based on the neutral, boilerplate, standardised language we aim at--nonetheless, the conclusion would be completely wrong"

If you mean, completely unique, such that a (long) text written by you can be uniquely attached to you and to no-one else? ... Possibly? If we found similar levels of specificity for everyone else... Maybe? That would require a ridiculous degree of specificity, and absolutely mind-boggling amounts of time.

These methods are used in cases where you suspect Text X may have input from Author A, but some from Author B, too. So, you gather some of Author B's texts, some of Author A's, and you compare them. Which measures are sufficient to separate them? Those are the ones you use. Enter Text X. Plot it against the clusters that are formed by Author A and Author B's texts. Where does it fall?

Point being: you really only look at texts and authors where you have reason to suspect shenanigans. Comparing text X with others until you find something similar (according to poorly-implemented tests) isn't likely to generate valuable author connections. Though stranger things have happened! Perhaps J.K. Rowling has published books under other names than JKR and Galbraith, and someone might find out some day!

I'm not sure if things like, say, boilerplate contracts (to take an example. I'm not implying you work in this field) with variation in some clauses where lawyers insert situation-specific agreements using standard legal language and phrases would show up all that differently. Or even tailor-made contracts which again use standard legal language and makes use of standard clauses. You'd be surprised at what an unrealized preference for certain determiners and punctuation marks can reveal. But texts like these are essentially designed to be indistinguishable from each other. And contracts go through various rewrites by different lawyers, too. I'm not aware of any studies here, but then again: stylometry isn't really my field, though some of my colleagues do research that could be classed under that header. I'm sure it would provide interesting challenges.

>249 Petroglyph: Yes, that's what I have done, compared the most frequent 6 words/letters in each text to the 6-most-frequent in each of the other texts in the corpus. As I also explained, I used phrases merely to show obvious patterns of only a single author using a given phrase and not any of the others; I did not compare phrases across all texts, but rather only the words and letters.

>258 faktorovich:
"I don't know why you guys keep challenging me without somebody stepping up to agree to accept the challenge on your end."

You are the one proposing a model. You are the one proposing a massive, unseen, gob-smackingly unexpected revision of centuries of authorship attributions. You claim your methods work.

The burden of evidence lies on your shoulders. It is not my responsibility (nor anyone else's) to prove your model right, except yours. It is not anyone's responsibility to prove not-your-model wrong. That is all yours.

Even if you, shouldering your responsibility, could prove that not-your-model was wrong -- that does not make your-model true. You'd have to demonstrate that separately.

Arguing against the status quo to such an extent as you're doing entails doing more heavy lifting than proving a smaller adjustment of the status quo. You have to convince, well, not just us anonymous LT users, but a scientific community.

This is your burden to bear. Don't complain that we don't share it. Because it isn't ours.

>262 Petroglyph:

Right, standardised writing is just a limit-case--my point was that completely ignoring context and relying only on analysis to assert how many authors are involved obviously CAN lead to wrong conclusions. But between that and unique styles (if such ever exist, i.e. if there really are any writers who write like no one else ever did or will) there's a lot of terrain where I imagine different people might share to various degrees either "quantifiable" or subjective markers of style, or both. Sufficiently so that a limited analysis might not reveal true multiplicity of sources.

Point being: you really only look at texts and authors where you have reason to suspect shenanigans.

That makes sense to me because it presupposes some basis for comparison. If we assumed for a moment total cultural amnesia, such that we lost all knowledge of authors and were left just with separate texts, it seems unlikely that a purely quantitative analysis could correctly re-establish authorship.

>241 faktorovich: I think you misunderstand me. Why would I want to read hundreds of pages of your writing when you haven't managed to successfully communicate your ideas either in the blog post interview or here?

I'm fully aware that the type of information and the amount of detail in a scholarly study is different than in a shorter summary for a non-specialist audience. Nevertheless -- as I'm sure you know, being a publisher yourself -- a clear and concise summary is an invaluable part of presenting one's work, and an absolutely essential skill in academic writing. It's how you (generic "you") convince audiences, who generally consist of busy people with a long list of potentially relevant literature that they need to decide whether to read, that your work, rather than someone else's, is worthwhile spending their time on.

By the way, the title of Pennebaker's book is not "Secrets" but rather "The Secret Lives of Pronouns", referring not to any incomprehensibility or secretiveness of his writing, but rather to his argument that analysis of inconspicuous function words like pronouns and articles can reveal unexpected information about the speaker/writer.

I'm guessing from your response that you aren't familiar with it; I mentioned it because it is one of the few titles I'm aware of on the topic of stylometrics that is written for a general audience. As I said, I don't agree with all of his arguments or conclusions, but I found it thought-provoking and thought it might be of interest to other readers of this thread.

>263 faktorovich:
Look. The most frequent words in an English-language text of any large size are bound to include these: the, and, for, to, a, was, .... Having many of these in your text is meaningless: everyone does.

It follows that combinations of the most frequent phrases ("I don't", "I can't", ...) are meaningless, too. They are merely combinations of frequent individual words. They are to be expected.

What is interesting is relative frequency: which words does this text / author / genre / whatever use more often than the baseline? Those are likely to be meaningful.

Even the tool for "unusual words" on one of your online tools requires you to paste in a "usual word list". "Unusual" words or phrases are those that are used more by a certain author compared to other authors. Do you even have such a list? Would you be able to generate one? I've linked to corpus analysis software in >219 Petroglyph: that would help you extract absolute word frequencies from a representative corpus.

Look. I pasted a link to a .txt version of Pride and Prejudice into your text analyzer. These are the top 6 three-word phrases:

i am sure (62)
i do not (59)
as soon as (57)
project gutenberg tm (56)
she could not (51)
that he had (37)

This does not mean that any of these phrases are diagnostic of Jane Austen! This does not mean that they are diagnostic of P&P! This does not mean that other texts that have these common phrases show Austen's ghost-writership. Look at them! They are combinations of some of the most common words (from the top 50, by the looks of things). It is exactly these kinds of phrases that are entirely expected in novels -- note the past tense (could, had). If this text had less dialogue, phrases such as "I am sure" or "I do not" would not nearly be as high. If Pride and Prejudice were written in the first person, that sixth phrase ("that he had") might have been replaced by a phrase containing "I".

That does not mean that Austen's signature style would have changed. Signature phrases, words, punctuation use etc. are the ones that unexpectedly common.

Also: I deliberately left in that project gutenberg tm. This shows you that phrases can be frequent, but not typical of an author. They can be frequent for reasons other than authorship. (A book with four volumes and, therefore, four occurrences of "chapter one" would not mean that that phrase is typical of that author!)

If we were to paste your comments in this thread (or one of your books) into a .txt file and feed it to that text analyzer, would the top 6 phrases be diagnostic of you? Go on. Do it. Tell me which phrases are "yours" based on one of your books.

I'm not denying that repeated occurrences of three-word phrases can be indicative of an author. Sure they can. But they must be three-word phrases that an author uses uncommonly often, as compared to a corpus of similar texts in similar genres by similar authors of similar classes, genders, ... Those are useful. Those can be diagnostic. Those can reveal author preferences.

First-person narratives vs third-person narratives will show differences in the "top whatever phrases". The point of identifying an author's style is precisely to look at things that are independent of first vs third person!

Now I know that you have not "analyzed" these three-word phrases while establishing your authorship re-attribution. In >263 faktorovich:, and in >119 faktorovich::

"The 3-word-phrases (the top 6 most-frequently-appearing of these out of all possible 3-word phrases in a given text) I pointed out as revealing in the Brontes' case are not one of the 27-quantitative tests involved in the basic method. I collected these 3-word-phrases for all texts and used them to find obvious patterns of phrases that only appear in the work of any given authorial-style. The use of a contracted vs uncontracted "I can't" is a significant stylistic divergence that is revealing when one of these appears among the top-6 most-common phrases because this means there are many instances of this preferred usage in the text vs. the alternatives. I did not use any 3-word-phrase patterns to establish attributions, but just used them in the writeups to show that the less easily understood mathematic matches vs. non-matches were also confirmed by these verbally-descriptive elements."

What you've done is a) run your tests, come up with some author profile. b) you've then looked at the three-word phrases from that author (or that text), and you have taken the most frequent phrases by absolute frequency as equally diagnostic of an author's style.

This is bonkers.

You've looked at the wrong thing. You've misinterpreted that thing, and attached much, much more importance to it than you should have. This is a mistake. Any conclusions you've drawn on the basis of these phrases has to be revised.

>250 melannen: A few days are not going to help me learn anything, if there are no steps you can think of to overcome the problem I explained I faced in using this software. There is no manual explanation for this problem in their system.

Here is a copy of an EEBO text - Percy's Coelia: https://quod.lib.umich.edu/e/eebo/A09307.0001.001?view=toc - this text is not available on Project Gutenberg; Renaissance texts that are on Gutenberg are mostly translated, and so are not comparable to original-spelling versions from EEBO, or useless for my experiment. EEBO is in the public domain and accessible to everybody for free. EEBO has plain-text websites that can be copied into Word files or other programs; the main text-processor I use just happens to be Word. Word allows for easy creation of plain-text files and other standard file formats (including UTF-8, but just not the type Stylo sees). No the standard file formats do not introduce any errors, unless the researchers has made a ridiculous bunch of errors in choosing a format etc. that generates errors. I just tried saving Coelia in Notepad as a UTF-8 and while saving the file in this format worked, Style definitively does not see/ recognize the UTF-8 format that was thus created. In other words, Stylo needs some other file format to work, and if you don't know what it is; then, you cannot have used this system yourself in application to real texts.

>268 faktorovich: It seems far more likely that there's a setup error with Stylo, rather than melannen having for some reason lied about the file format needed for Stylo to work. Are you sure your corpus folder is in the right place (as explained in >251 Petroglyph:)? That's the first thing I would check, if Stylo isn't "seeing" the files at all.

>251 Petroglyph: I changed the folder name to "corpus" and Stylo still did not see the files in it in the UTX-8 format. Now you are adding a step to your method that makes no sense with the previous steps you listed. You said that I just type "Stylo" etc., and then open any folder in my computer where the corpus is. Why would there be an option for me to choose a folder, if the system only recognizes files that are in a specified directory? I tried typing "getwd()" in Stylo and in my computer's command options and nothing happened. You want to create a new set of steps that includes placing files in this specific directory, and how to find them etc.? Otherwise, nobody can follow existing steps to get any kind of results with this method. And then you are adding new steps in your last paragraph that you are now saying are needed to get statistics that were not in your initial list, and these steps cannot be applied without far more precise steps as to where/what is being chosen, and how to generate these components. Your method is not practical and the glitches we are discussing should not be there if this tool was as popular as you have claimed.

>258 faktorovich: >I don't know why you guys keep challenging me without somebody stepping up to agree to accept the challenge on your end.

Hey, I'm going to be really frank here: nobody is taking up your latest challenge because they don't have any reason to.

Firstly, because almost nobody in this thread is really that invested in whether your results are true or not? Some of us are passionate about the idea of outsider science in general, and it makes us sad when someone presents themself that way but has clearly not done the basic foundational work to know what they're doing. Some of us are just unrepentant pedants who can't stand someone getting even the smallest facts wrong without correcting them - and you're getting a lot of small facts wrong. Some of us are just enjoying a good internet argument about something with no stakes whatsoever - it's a nice change. Some of us don't like the idea that LT gave their platform to research that doesn't stand up to fairly basic scrutiny - but that's a problem we have with LT, not a problem with you; nothing you do can fix that problem because it's not a you problem, it's an LT problem.

None of those reasons are the sort of reasons that would make us want to test your method, because none of them really have anything to do with whether your results are valid or not. We would still have the same problems with what you are saying here even if your results were proved 100% correct, because the problem is not your results! It's how you're going about backing them up, which is poorly, and with clear evidence that you don't even understand why you're not backing them up well.

Secondly, because people in this thread have taken up your challenges. You challenged me to give you simple instructions on how to do a reproducible test using Stylo. I provided them. You followed them halfway, then stopped following them and tried to substitute your own data (which wasn't covered in the instructions) and claimed that meant I was wrong about being able to write simple instructions, because my instructions didn't work when you changed some things. This is a super common pattern with people who try to present unusual claims - they will challenge their opponents to do something, and when the opponents do that thing, they "move the goalposts" and present a new challenge which they then claim nobody will try. It's really easy for someone who has seen it before to notice when someone is using this strategy (you are, if you hadn't noticed!) and that makes it obvious that there's no point in undertaking the challenge, because no challenge would ever be good enough to settle things, there will always be a change snuck in or a newer, better challenge.

Thirdly - it's work! I wrote out those instructions because I wanted to play with R and it motivated me to do that. But something like trying to reproduce your whole experiment would be a lot of work, as I'm sure you know! None of us here care enough about attribution of early modern english texts to want to put in the work you've put in - you're the one who's passionate about it! I think the topic is interesting enough that I'd really like to see you approach it with better methodology, but if I'm going to do that much work on stylometry and ghostwriting, I'm going do Stratemeyer books, because those are the ones I care about. Petroglyph has actual paid work they need to be doing instead. Testing your work isn't our job.

>270 faktorovich:
What is your working directory? Type "getwd()" (without the quotes).

Make sure your "corpus" folder is in that working directory. It should only contain the files you want to look at. Preferably al in .txt format.

Are the files you've saved in .txt? UTF is a character encoding used in plain text, so your files should be .txt.

Don't uses MS Word (or libreoffice or google docs or whatever) as an intermediate. These apps save texts in other character encodings, which may mess with the UTF-8 encodings as you save them as .txt files.

If you copy straight from web pages, please copy straight into a notepad file. Then save as .txt.

If you must use other formats, you can. Look at the first tab when you run Stylo to see what it accepts. It accepts html files.

On this page, I rightclicked on the html text link and selected "save as", then saved as html page. Not html, complete -- that saves a folder with images alongside the page. Just "save as", then select "html only".

I did the same on the equivalent page for Sense and Sensibility.

If you paste these two files into your corpus folder, and then have stylo run, you can select (on the first tab) the option .html, instead of .txt. Running a PCA with all the defaults on those two files gives you this:

>271 melannen: Very nicely formulated, IMHO.

I like mathematics and I have quite a few math books including some rather silly ones. One of them is written by a medical doctor (x-ray specialist) who wrote a book about patterns and prime numbers. It got a rather unfriendly review by the readers and one of them linked to this page: https://primes.utm.edu/notes/crackpot.html
Please notice that I'm not trying to insult anyone but just pointing out why announcing unusual results is risky business for anyone. Should I one day wake up and have a proof of the ABC-theorem in my head, I hope I'll go through the crackpot.html check list before calling national media.

ETA: A bit of the x-ray doctor's stuff: https://mersenneforum.org/showthread.php?t=6409 which triggered the crackpot alarm.

>273 bnielsen:

this page: https://primes.utm.edu/notes/crackpot.html

Okay, that page is awesome. And nearly all of it applies in fields other than mathematics. Love it.

>273 bnielsen:
Neat!

>260 norabelle414:
Cheers!

One of the problems with letting Microsoft Word anywhere near a text for machine processing is the default behavior of converting "straight quotes" (ASCII 22 ") to typographers' or "curly" quotes (“ ”). This also applies to apostrophes dashes etc.

These can be saved into a text file with a UTF-8 character set but the two types of characters are not equivalent to the computer and will confuse software trying to process it.

You can turn this behavior off but most do not. When I have been involved in web applications there has been no end of problems with users doing a copy-paste from Word because of these defaults. Often it is necessary to come up with a series of search-and-replace routines in the code to catch them to achieve a "paste from Word" function. Yet, whatever you think to include, there will be some other one that will not be caught. It is like a child's game of Whack-a-Mole.

Even pasting to Notepad or most text editors will not help because the characters are legitimate but they are different.

Software like BBEdit on the Mac can run a routine to "straighten quotes" but this doesn't touch the em-dash, etc.

James

I saw some reference to post-1926 texts being off limits for the study. This sounds like a simplified rule concerning U.S. copyright law.

As with most things, the reality is more complicated.

As I recall, the 1927 texts don't become public domain in the U.S. until 1 Jan 2023. Thus, even the 1926 texts would not be available until 1 Jan 2022.

Up to a certain point there was an opportunity to renew copyrights beyond the first term of 28 years for works for hire under the 1909 law. In the year following the 28th after copyright and publication, the owner (publisher, author, heirs) could renew the copyright and get another 28 years of protection (56 total).

Extensions to the Copyright laws in the 1970s and 1990s have taken us to our present level of 95 years for works for hire. I find them to be absurdly long in an era when there is pressure to make 20-year patents shorter.

The result of this is that for works first published between 1923 and 1963 that if they were not explicitly renewed (with a record that can be found from the Copyright Office), the work is now public domain.

This can happen with publishers that went out of business like Saalfield who had the copyrights for their books but were not around when the renewal time came up. Some authors and even publishers were sloppy about renewing their properties.

So, in an effort to prove a negative, you look at databases like the Stanford Copyright Renewal database (based on the initial efforts of a Google engineer to accurately digitize published renewal records). It usually pays to search several ways to see if a work was renewed. If it was not, there is an excellent chance that it is now public domain.

Using this method, I have been able to convince groups like HathiTrust.org or Google Books to make certain works available in full view on their systems. Once this is done, it is usually a matter of time before they are made available on Gutenberg.org for a text that might be suitable for textual analysis through stylometric software.

Thus, there may well be texts after 1925 that are available for analysis.

My field of specialty is juvenile series books and I am writing an extensive Series Book Encyclopedia which deals with nearly 1,300 such series and thousands of volumes. I have noted which of the 1923-1963 works appear to be public domain since this provides a clue about which works might be appropriate for digitization projects.

James

>268 faktorovich: If you can't learn things, there's no point in any of this. A few days won't make you an expert in a new piece of software - but they will give you time to work through enough tutorials and free video courses to know where to start. (Like I said! I wrote those instructions with half an hour and google! I am *not* a stats expert - I'm just someone who is willing to try learning things, even things I don't immediately understand.)

>>No the standard file formats do not introduce any errors, unless the researchers has made a ridiculous bunch of errors in choosing a format etc.

I am trying to tell you that somebody using Word to prepare text data for statistical analysis *has* made an error in choosing a format. Don't use Word for anything other than preparing a text for printing! It's not good at other things! More importantly, using it is going to broadcast to anyone who does this work that it has ridiculous errors!

I didn't go into detail about text encoding (which is different from file format) in my original instructions, because that wasn't the goal of the instructions - the point was to demonstrate that anyone could do a simple Stylo run in fifteen minutes from scratch, with basic instructions. I'm not going to go into detail about text encoding here, either, because that is basic stuff anybody working with digital text files should know, and, again, if you're willing to try to learn stuff you don't know, it's easy to find internet resources about it. (Is it simple? At the beginning it is, but it can get super complicated. However it's not something you can overlook because it can matter a lot to how valid your results are! If you'd come at me with justifications of why you'd chosen word and how you'd carefully compensated for its shortcomings, I would be with you all the way. But you didn't, you came to me not even knowing what UTF-8 is.)

I am going to try to figure out why your Stylo test isn't working though because I'm a sucker for an unsolved problem.

(First off, the texts on that website aren't plain text. They're xhtml/sgml. It says that right on the "About" page. Your web browser makes that xhtml look like formatted text, because that's what web browsers are for. When you copy it and paste it somewhere else, some of the information in the xhtml about how to display the text is going to change, because it won't be xhtml in a web browser anymore. Some of those changes may make your stats results inaccurate. But that's all beside the point really.)

Anyway, I copied-pasted the Coelia text to Notepad and saved it as Coelia.txt, which made it a UTF-8 txt file. And then I followed my instructions 4-8 exactly. It gave me an error because the point of that test is to compare texts to each other, and you can't compare one text to itself. Is that your problem? I copy-pasted Faustus from that website too, saved that to the Corpus folder from Notepad, and tried again, and now that it had two things to compare, it worked.

If you had two or more .txt files to compare, maybe the problem is that you had other files in the folder as well? My instructions didn't mention that you couldn't have any other files in the Corpus folder, because they told you to create a new folder from scratch. If you have any files in Corpus other than .txt UTF-8 files, delete them, and then try again.

If neither of those things work, I can't say. I can say the problem isn't the way you're converting to UTF, because that works for me.

(Stylo doesn't actually require UTF-8 by the way, That's just what it assumes you're using if you select "plain text", because it's the most common plain text format. If your originals are xhtml files, then you should select "html" instead. The options are right there! It does require you to know the difference between html and plain text, though.)

>278 melannen:
👍

>254 LolaWalser: The description of "Exercises" explains that it varies genres such as the "sonnet" and "mathematical formula". Genres are entirely conscious choices and can obviously be manipulated by a scholarly writer who is familiar with their formulas. I don't know why the author has called this "style", but perhaps something has been lost in the translation. There is no question that authors intuitively have an aversion or an attraction to exclamations, and have rhythms of punctuation that feel natural to them while others don't. The patterns are only entirely unique when a large number of different elements is measured as is the case with my 27 tests. While even the exclamation rate can vary between texts the combination of multiple elements is still going to register the underlying author's signature. If you follow the simple steps I provided, your conclusion will be entirely right, unless you make unpredictable selection-bias choices in building your corpus or make mistakes with the process. Let me know if you want me to look over your data after you attempt an experiment.

>277 Keeline: Using this method, I have been able to convince groups like HathiTrust.org or Google Books to make certain works available in full view on their systems. Once this is done, it is usually a matter of time before they are made available on Gutenberg.org for a text that might be suitable for textual analysis through stylometric software.

With my English Wikisource admin hat on, may I point that we're here too, producing transcriptions of works from HathiTrust and Google Books?

>264 Petroglyph: You are saying that the history of Britain is my burden alone. Not you, or any other scholar, but me specifically. I agree. Absolutely. I am working on testing the set of texts you ran through your experiment, and I will post the information I gather when it is finished. I am just waiting for the software to finish processing, and thought I'd respond to a few other posts.

>265 LolaWalser: If you only test word-frequency or any other single measure, it will become difficult to distinguish between styles. But fingerprint analysis works because there are many ridges and curves and other unique features, all of which jointly create a unique set of features that is extremely rare and appears in as few as only one human on the planet. This is why I use 27 different tests in combination. We should measure more texts with computational-linguistics, and we surely should not only test the mystery-texts without comparing them with a significant number of other texts in the surrounding canon to check if other texts might not seem like mysteries, but are, or who out of all likely authors is the real underlying author. Instead too many computational-linguists start with an assumption they have guessed who the author is and only test these assumptions on a handful of texts/bylines.

>283 faktorovich: Looking at more information is fine, but calculation without calibration is always a danger. As I said, it's easy to use calculations to establish that every one of those 284 works has a different style and hence a different author, or different calculations to show that every work is written in the same style which can be labeled as English (as opposed to those who write in a style known as French or Spanish or Estonian or Swahili.)

If you assume that most of the works are correctly labeled and are trying to figure out certain anonymous works, it's easy to check that your calculations are producing reasonable answers. The further you go away from that assumption, the harder it is to say anything. If you know nothing about the authors or the works, >253 Petroglyph: could be based on the works of 16 different authors or 2 or 1, depending on how much distinction between the styles you think differentiates an author.

Computational tools can't tell how much variation you can expect from a single author, or how much difference you can expect between author's styles, or even if they're a reliable tool for separating authors at all. And it could very well be different for Renaissance English than it is for modern English, given the reduced cultural variation and different literary forms. When you say there's six ghostwriters, that's purely bias; someone could look at the same data and come up with more authors, or possibly even less. Like many have done before, you're tossing around numbers and coming up with answers that match your bias, and not admitting (or possibly even understanding) the role your bias played in getting the answers you got from the numbers you got.

You say your data is available on github, but what really matters is your code. I don't know anything about that period of English literature, but I've been doing data science since before it had that name, and can comment on methodology. In particular, I'm interested in:

* Is this a clustering mechanism where the number of clusters (authors) is unknown and the algorithm seeks to simultaneously determine the optimal k number of clusters and to attribute individual works to a particular author?

* If so, what is the mechanism used for determining optimal k? That's one of the more complex and nuanced parts of unsupervised clustering algorithms, and is critical for your situation to get right.

* How does your methodology perform against texts of known authorship, particularly when an author's style may evolve over time? Does it correctly attribute early and late works of the same author to a single author?

* Relatedly, does your methodology correctly distinguish between authors of texts of known authorship when they're known to have been in close communication with each other?

* How do you construct your feature sets? Why do you select the features you did? Clustering on categorical as opposed to numeric features can be very challenging - what is your distance metric?

>285 lorax: "Relatedly, does your methodology correctly distinguish between authors of texts of known authorship when they're known to have been in close communication with each other?"

Y'mean like the Brontes, who were writing together since before they were ten?

>285 lorax:
Excellent questions, all!

susanbooks (#286):

Bingo. But for other examples. If it can't distinguish between authors who began corresponding as adults, how could it possibly distinguish between siblings?

>280 faktorovich:

I don't want to belabour the point with a single example, I was merely addressing your remark that

quantitative linguistic features are unconscious usage patterns that the writer cannot deliberately control.

Queneau does deliberately vary features mentioned as "quantifiable" markers of style (not just genre or tone). And, while this book in particular is based on a short text, there are other longer, even novel-length examples of Oulipian literature produced with constraints that result in what I suppose one might call artificial style, different to those authors' unconstrained writing.

In short, I take your point that there are features we reproduce in our writing "unconsciously", but I'm not convinced that they can't be or often aren't manipulated at will.

I have carried out a verification-check titled “Lunch Test”. I have added the above diagram and the full data table for the raw numbers and the attribution steps to my GitHub: “Lunch Test - Attribution Diagram.jpg” https://github.com/faktorovich/Attribution/blob/master/Lunch%20Test%20-%20Attrib... and “LibraryThing - Lunch Test - Data - 12-9-2021.xlsx” https://github.com/faktorovich/Attribution/blob/master/LibraryThing%20-%20Lunch%.... This test re-affirmed the earlier findings of my “Koppel Experiment Reviewed - Data Tables.xlsx”: https://github.com/faktorovich/Attribution/blob/master/Koppel%20Experiment%20Rev.... There are only two linguistic signatures between the three “Bronte” sisters, and since only one text from “Emily” could be tested, I have assigned the byline for this linguistic group to the multi-texted “Charlotte”. “Anne” and “Charlotte” clearly have distinct signatures, as do “Austen” and “Corelli”. There were a few linguistic intersections between all of these bylines that indicate occasional co-writing of some of these texts by the authors in these distinct linguistic groups. One such example is “Austen’s” “Northanger” that is similar to “Anne’s” “Agnes”. The most significant example of this cooperation is between Corelli texts and a few “Anne” and “Charlotte” texts. The data could not establish if any of these bylines are genuine or if they represent ghostwriting-purchasers/pseudonyms or the like. It only establishes the degree of similarity and difference between these texts. This test confirms the accuracy of my 27-tests computational-linguistics method because nearly all of the texts correctly show similarity within a given byline, and divergence between that byline and the other bylines in the corpus. If the results were random, there would not have been any noticeable pattern in the data. Most of the 10+ test matches are correctly located on the data table within the same bylines. Thus, when these matches show up in other bylines as well or when two bylines match each other’s texts, these byline-breaking matches accurately indicate two or more collaborating signatures, or a single signature using two or more pseudonyms.

In addition to this summary table, the full data set includes the relative word-length of these texts, the year when they were published and other data to assist future researchers. It also includes the exact numbers of exclamation marks and the other test results that contributed to these matches/non-matches. It also includes the data on the top-6 most-common letters, words and 3-word phrases in this corpus. As you can see from this information, this group of texts are unusually similar to each other, as 9 out of 16 of them share the same b-pattern of top-6 letters (e, t, o, a, n, i) – all of Austen’s 6 texts use it; and 9 of them share the A-pattern of top-6 words (and, the, to, I, of, a) – including all 5 of “Charlotte’s” texts (including the “Emily” likely-pseudonym). There are also a strange number of texts that share the same most-common 3-word phrases: 3 with “I am sure” (preferred by “Austen”), 4 with “I could not” (mostly preferred by “Charlotte”), 3 with “I do not”, and 3 with “I don’t” (the latter 2 phrases are shared by “Austen”, “Corelli”, “Emily”, “Anne”, so they are likely to belong to the main collaborator in this group). The presence of these byline-specific letter, word and phrase preferences further confirms the quantitative attributions in this experiment.

The broad conclusion of this test is that the current accepted byline-attributions and the belief in these authors’ purely independent authorship is erroneous. I found similar collaborative and ghostwriting in male bylines from these decades in the “Koppel Experiment”, so this is not a statement about female authorship as opposed to male authorship. This is simply an honest representation of professional writing in the highbrow canon we have been learning in school. If any of these female bylines were actually men working under female names to attract readers, this means that actual women might have otherwise found these publishers who were aiming to tap female-readership. If a man was paid for the writing-labor credited as the “work” of a glass-ceiling-breaking “woman”; retaining his byline on a book under the assumption that the existence of this byline empowers all women is faulty. Then again, the underlying ghostwriters could have been women, but only a testing of hundreds of bylines from this century and a full study would uncover the precise truth. This experiment simply distinguishes the linguistic signature in this small sample corpus.

So the only possible and reasonable explanation for a similarity between texts attributed to two different authors is collaboration, or else some hitherto unsuspected ghostwriter must be responsible for both?

BTW, I'm bad with dates, but a quick google search indicates that Marie Corelli was born in 1855; Charlotte Bronte died in 1855, Emily in 1848, and Anne in 1849. Jane Austen died in 1817, when the oldest of the Bronte sisters was only a year old.

Yet you think it is plausible that they collaborated with one another? Or that there was some ghostwriter of such longevity that his work spanned several generations?

Alrighty then. Carry on.

>290 faktorovich: One of the key characteristics of science is falsifiability. As far as I can tell, there's no set of results that could be produced by your methods that you would consider evidence that they're wrong. If your test disagrees with well known facts, then you've felt free to dismiss those facts as wrong.

>291 spiphany: To answer your question, I organized all of the texts in the table by their publication-year. The 5 earliest-published texts match the "Austen"-signature, with only 1 "Austen" text first-published in 1871, though Austen had died back in 1817. One of the "Austen" overlaps - "Anne's" "Agnes" - was published in 1847. If you let go of the idea that either "Austen" or "Anne" had to be the authentic authors behind these bylines and consider that a ghostwriter could have started writing under the "Austen" byline in the 1810s or earlier and kept writing after Austen's death into the 1840s or later; then, there is no abnormality in this data. The collaborative match between "Corelli's" "Young Diana" (1918) and "Charlotte's" "Shirley" (1849) suggests that Corelli was definitely not the underlying ghostwriter who wrote under her byline, as Corelli was born in 1855; Corelli happened to be born under a different name that she changed to the pen-name, so this begins to re-affirm the ghostwriting conclusion. It is likely that texts were written close to the year when they were published, and in this corpus a few of the texts were first-published after their byline-author's death; therefore the claim that the byline-authors were dead or not yet alive when the first-texts re-attributed to them were written only confirms my findings.

A match between 2 different bylines can mean a few different things: 1. collaborative writing between them, 2. single ghostwriter/author writing under 2 bylines, and a few other scenarios. The number of matches within the byline and a few other elements decide what scenario is the most logical for a given couple of texts.

I deal a lot with the complex roles of ghostwriters in juvenile series books of the 20th Century. There are cases where there are multiple "authors" in a given published text and it is nearly impossible, without an edited manuscript, to learn who did what.

For example, the Stratemeyer Syndicate provided its ghostwriters with an outline of a couple to dozens of pages from which they produced a book manuscript of a couple hundred pages. When the writers transcribed content from the outline, the style should be more like the author of the outline. After a manuscript was turned in, there would be rewrites by one or more people as part of the editing process. It is easy to have three or six "authors" in one of these books.

I am prepared for the comment of "how they did things in the 20th Century is irrelevant to earlier periods." I disagree. Further, just how professionalized are we to believe that authorship was in the 1500s through the early 1800s to have ghostwriters working everywhere and just a handful of them? It just is too much like a conspiracy theory to be believable without some very solid proof, both intrinsic and extrinsic to the texts.

With something like the Brontë sisters acting as a writing community and forming a read-and-critique circle, wouldn't the suggestions by the others influence what was published.

Also, the usual notion is that female writers adopt male or gender-neutral (e.g. initials) pseudonyms to increase readership (e.g. fantasy and sci-fi genres). However, you are saying that some male ghostwriters used female pseudonyms to increase readership. Why were the earliest versions published under male names which are now seen as pseudonyms?

James

This thread is the gift that keeps on giving.

"If there's nothing wrong with me, there must be something wrong with the universe."

>253 Petroglyph: As I explain in a separate post, I have performed this experiment using my 27-tests method. I used the same plain-text files you posted the link to. I am now going to explain why your findings are faulty in comparison with my own. The first problem I have noticed is that your diagram uses "Classic Delta distance" as a measure, but you do not define what this is a measure of precisely. This is what I found about this concept on a brief search: https://rdrr.io/cran/stylo/man/perform.delta.html - Stylo does not specify which specific "measurements" are included in this calculation, so it can be anything. What does the spectrum between 2 and .5 mean; these numbers do not make any sense without non-matches for reference.

My data also contradicts that "Susan" is significantly less like "Emma" and still further less like the other 4 "Austen" novels. "Susan" has 7 matches to both "Emma" and "Northanger", and "Susan" is most like "Mansfield" that this diagram shows instead to be furthest from it. Both "Susan" and "Mansfield" share the b-letter-pattern (e, t, a, o, n, i). The first-person in "Susan" is reflected in the top-6 words measure because "I" (as well as "her") is present among its top-words, whereas "Mansfield" does not have "I" among the top-six (and only has "her"). Because the method described by Petroglyph uses word-frequency as its main linguistic test, his attribution conclusion regarding "Susan" is heavily influenced just by this first vs. third person difference. Because word-frequency is only 1 out of 27 tests that I use, this single measure did not influence the overall attribution of "Susan" as similar to "Mansfield" in my results.

However, it would be an error to subtract most-common words from consideration all-together because both "Susan" and "Mansfield" still shared the "her" pronoun that this author clearly favors. In most texts I have found very useful in revealing patterns among the top-6 words, and they more frequently confirm the overall attribution than contradict it. With this test in the mix, researchers can see what influence voice-change had on the linguistic results.

Petroglyph's "Cluster Analysis" visual of all of the texts compared to each other also re-affirms my finding regarding "Emily's" style being similar to "Charlotte's"; his visual just happens to place it on a slightly divergent bend, and thus Petroglyph sees this as sufficient to say that "Emily" and "Charlotte" form two distinct signatures, and not variations within a single authorial-signature.

The biggest problem to Petroglyph's entire "Lunch" experiment is that he did not even include a link to the raw numeric data, and just shared these visuals. He is not sharing exactly what features were measured; he needs to list these, so I can double-check why the elements he measured yielded these results. In contrast, all of my data-set is available, and anybody replicating the steps of my method can check the accuracy of my process, and trace-back the reasons for the summary results.

"Diana" is indeed a bit different from the other 2 "Corelli" texts but there are many other significant elements to consider aside for the first/third-person voice, including "Diana's" publication decades after the other two. "Diana" has a different letters and words pattern from these other two novels; but it also shares phrases such as "one of the" with them. "Diana" actually has the "she" pronoun among its top-6 words, whereas the other two include "I" in the top-6; this makes me question if Petroglyph might have confused the first and the third-person voices between these novels. On all 27-tests combined "Diana" is not significantly different from the other two.

My findings also contradict that "Anne's" novels are similar to "Susan", as they are a non-match at only 5-7 tests in common. For example, these three texts have different top word patterns, as one favors "I" another "her" and another "you". Without the raw data, I cannot research what this testing method could have found to be similar between them.

And Petroglyph has repeatedly said that he removes the most-common words from consideration to avoid the bias that comes with first/third-person/ generic variation, but in contrast to these claims he is saying that these pronouns have had an attribution-changing contribution to these findings. This would have been the case if he just tested for the most frequent words in these texts (only 1 of my 27 tests).

My test took me a day instead of just lunch-time, but the data it provides proves the case with overwhelming evidence; whereas Petroglyph's method just plots relative similarity points or a single data-output conclusion. He is not even showing all of the text-to-text comparisons or the degree of similarity between them, and instead is showing their relative linguistic values. If readers trust these conclusions, these are nice visuals; but anybody who doubts these conclusions cannot check them to see what might have led to misattributions. The 35% and 16% figures he gives are entirely fictitious and there is no explanation what they can be based on; they just appear to be used to make the whole thing seem more detailed than it is.

Overall, Petroglyph is reporting similar cross-byline similarities that I am with my method; but he is denying that these similarities indicate the presence of pseudonyms, collaborative writing, ghostwriting etc. What this data is saying is not debatable. If there are cross-byline similarities; it cannot be true that all of the authors are only writing under their own "true" bylines.

>276 Keeline: These various replies regarding UTF-8 working in Stylo with specific adjustments all really mean: there are an enormous amount of many fine-point-problems that have to be known to perform the basic step of accessing a folder with the files to test. There are not only no manuals that address these possible glitches, but even specialists do not know which of these potential problems can be plaguing a given user. All this just means that Stylo is not usable by the public, either intentionally or because the programmers did not have time to write up a coherent manual.

>278 melannen: The texts I used were UTF-8. Are you saying Word cannot create a UTF-8 file, even if it has an option for this type of plain-text file creation. Your responses are nonsensical. A set of simple steps for any method means a set of steps the user can follow to perform the test. If you think everybody should know advanced programming before they can use your simple steps; then, they are steps meant for advanced programmers, and not for the general public. You have been asserting that your method is accessible to the public, so you have to choose if your method is too complicating, or too simple for either of these arguments to make sense. I did not download texts from any website in xhtml/sgml; I have no idea why I would have done this absurd error. As I explained before, I tested creating a txt file in Notepad, and it was also not recognized as an acceptable file-type by Stylo. Stylo did not recognize other types of plain-text either when I tested it.

>284 prosfilaes: There is no bias in my re-attribution of the British Renaissance to 6-ghostwriters. I am simply reporting what the data says. You can look at the exact data on my GitHub to check this conclusion. Nobody has ever compared 284 X 284 texts against each other in this Renaissance corpus, so there are no rival arguments about this group of texts as a whole. All previous studies have been about small(er) groups of texts that only tested single elements such as word-frequency and thus they have not tapped into the underlying linguistic patterns I describe in my book.

>285 lorax: You have not read the interview with me this discussing string is about. I explain the steps involved in my process there. They are steps that anybody can do with their home computer and free software; they do not involve any coding. Yes, I cluster authors together, but based on the degree of similarity or divergence they have with other texts. The main difference with my method is that I do not assume any bylines are accurate, and thus do not start with byline clusters, but rather check which texts are similar to each other and let their similarity determine they belong in a single cluster. The point of my findings is that the concept of "known authorship" is faulty. The byline on a text might or might not reflect the true underlying author behind that text. For example, more than half of the texts in the Renaissance were originally anonymous, but most of these now have bylines that were assigned to them by scholars across the past 400 years; these assigned bylines are considered as "known authorship". Meanwhile, some texts that initially had "Shakespeare's" byline have been de-attributed to other bylines by scholars; and these other bylines are now "known authorship". Testing if an attribution method re-affirms "known authorship" is like testing for a murderer by considering all outcomes that lead to anybody other than Suspect X as faulty methodology. The features I use and the data related to all of them is included in the spreadsheets I have made available for public viewing/ testing on GitHub. I measure divergence or "distance" by the number out of 27 tests two texts match on, so this is a very complex and thus accurate 27-level metric.

>292 prosfilaes: "Falsifiability" is not a "key characteristic" of all science. The gravity equation g = GM/r2 might be something that cannot be proven to be wrong; this does not mean that gravity does not exist, or that this equation must be unscientific.

>290 faktorovich:
Alright, let's take this at face value for a second.

• What is the cutoff for similarity? The 10+ that you mention is clearly supplemented by 7 through 9 as well. In your spreadsheet there's several values of 7 (several in the Jane Austen block, CBronte block and Corelli block) that are judged as "similar". But a score of 7 between Tenant and Lady Susan, between Tenant and Mansfield isn't. There are multiple instances of 8 and 9 that aren't highlighted as co-writing similarities (such as between CBronte and ABronte). What is the rationale behind allowing some of these to "count" but not others?
  
  
• A single individual behind both Charlotte Bronte and Marie Corelli? This needs no comment from me.
  
  
• Posthumous publication of an unfinished book by a world-famous author? Naaaah! Impossible! Co-authorship!
  
  
• In this "forthcoming article" of yours, did you already note your "similarities" between CBronte and Corelli? Were you aware of them? Will you incorporate this new information into your article?
  
  
• If an author like Corelli uses language and speech patterns that were old-fashioned and even archaic for her day, would that not account for similarities between her texts and those from thirty, forty years before? Why posit co-authorship?
  
  
• How do you distinguish between actual similarities and coincidences? Books authored by the same individual vs different people with similar styles. If you keep looking for similar author signatures, you are bound to find similarities that are due to chance and not shared authorship. How do you distinguish the two?
  
  
• Logically, if you have 27 tests, each of whose results you manually alter to 1 or 0, you have a total of 729 potential author signatures that your method can recognize as similar from each other (27^2). There are only 729 combinations of one-or-zero-scores that you can add up to the number you put in this table (7, 10, 26). Well, you add a test, so the max is 28. Even so, that means 28^2 = 784 logically distinct possibilities. So. There must be a number of authors whose patterns match up but who could not possibly have produced the same body of texts (because they were born 150 years apart, or sth).
  
  
• I've already explained in >267 Petroglyph: why the "top whatever words" are no corroboration of anything whatsoever. So that takes 6 tests out of your 27. The same thing goes for the "top 6 letters" (It is not unreasonable to consider the possibility that a novel about the Dashwoods may show different letter counts than a novel about the Bennetts. You need a way of preventing such incidentals from measurably impacting your diagnostic features, and character names and locations may impact those raw frequencies. Also, Have you removed all "chapter one" etc from your texts? Those will impact counts, too!). So there go another 6. That leaves really only 15 tests. 15^2 = 225.
  
  
  No wonder you keep re-attributing lots of authors to a tiny number of "ghost-writers".

>301 faktorovich:
Well, after Einstein and Galileo, we're on to Newton.

>294 Keeline: There have been known writing Workshops that used pseudonyms or a communal name from ancient Judaic literature, to Renaissance Italy, to the 20th century as you point out. There is nothing unique about the 20th century or the Renaissance that would make the case that ghostwriters were working during these times any more or less believable. The central distinction about the British Renaissance might be the small number of these ghostwriters (6), but this number makes sense when vagabond/ fraud/ monopoly laws I explain are taken into account. They could not really advertise or even privately solicit for new ghostwriters to join the group without facing fraud etc. charges. Imagine you were a ghostwriter/ publisher who worked with ghostwriters: you might slap semi-random pseudonyms (male or female) on minor publications that you don't think will be very successful; but if a marketing ploy of using female names comes up later on that gives a set of books an advantage, you would leap bylines. Examining existing records with these possibilities in mind is likely to lead to very different conclusions, than if a researcher accepts the "Bronte"-bylined handwritten manuscripts as unquestionably "known" as their own, or without considering the possibility they can be forged.

>302 Petroglyph: The gray boxes I entered into the table are designed to assist the viewer with seeing the overall similarity between the texts in the cluster. There are degrees in their similarity: a higher similarity means sole-authorship, whereas a lower score can mean co-authorship or that it is indeed large the work of another author. 7-9 is the gray-area; in some cases two texts might have a 7 when X is compared to Y, whereas when the order is reversed they might have 9 tests in common; thus, it would be unfair to disqualify all 7 matches without considering if a given text matches other texts in the byline being compared against etc. You are fixating on these gray areas, when most of the table shows far less than 7-matches in non-matches or in the parts of the table that are not highlighted in any color.

I did not say there was one author behind "Charlotte" and "Corelli" - I said there was a collaborative author who helped both of these bylines separate authors - this collaboration is the reason these bylines register as largely different from each other, but with some overlaps between them that show collaboration, which could have been one of them helping the other one, or a third collaborator intervening to help both of them.

Yes, in my study I have found that most posthumous books are ghostwritten or share linguistic styles with otherwise-bylined texts that were written after the bylined-author's death.

I did not analyze Corelli in my forthcoming Journal of Information Ethics article; I did an entirely new study just for you. The findings on Corelli do not add anything new to my analysis, as I discuss 21 texts in that article and there are several other mysteries that I solve that are more interesting than the Corelli case.

"Old-fashioned" language? Just like comedians can mimic accents and "old-fashioned" language, so can most professional writers.

The more I apply computational-linguistic tests to texts and research these authorship mysteries, the more convinced I am that there are no coincidental matches; all matches indicate shared authorship. You can call this similarity editorial-input or the like, but if most of the text is the original work of this "editor", he or she is the true underlying ghostwriter.

The 729+ possible combinations you refer to is the reason my method can distinguish between the degrees of similarity. Obviously, my method is not equivalent to the precision of handwriting-analysis or DNA, as it is far more likely that two humans in the world have similar writing styles than 2 have identical DNA. However, all humans on this planet do not write books professionally. To make a living from writing, the truth is a ghostwriter has to write constantly. These top-ghostwriters/writers have the stylistic patterns that are of interest to researchers. According to my findings, most ghostwriters, publish at least one text with their own byline, which makes it possible to identify them as the underlying author. Renaissance documents have been digitized and made publicly available in a way that makes it possible for a researcher to check documentary proof vs lack of it for the existence of a byline to separate pseudonyms from real people etc. I have already considered nearly all of the bylines on all published texts from the century of the Renaissance before reaching my conclusion about the 6-ghostwriters and naming them. This is why I have proven the Renaissance case in my series beyond reasonable doubt, whereas any case I make about the Brontes in these mini-experiments are surface glimpses into their authorship attributions.

The top-6 letters are only 1 and not 6 of my tests, as is the case for the top-6 words: these are just 2 out of my 27 tests. If you actually attempted to follow the steps of my method to see how it works, you would not make this error.

>296 faktorovich:

Here we go.

The first problem I have noticed is that your diagram uses "Classic Delta distance" as a measure, but you do not define what this is a measure of precisely.
Burrows' Delta. Hover over the various options in Stylo. It'll tell you what they measure. (If you want to know more, check this overview paper: https://doi.org/10.1093/llc/fqx023)

Or you can take it from the horse's mouth: Burrows, J. F. (2007). All the way through: testing for authorship in different frequency strata. "Literary and Linguistic Computing", 22(1): 27-48.

You need to look this up?? It's a bog-standard method!

"Stylo does not specify which specific "measurements" are included in this calculation"
False.
I will just repeat what I said in >251 Petroglyph: Stylo "will also dump into your working directory a word list of all the tokens in your corpus; a table_with_frequencies.txt, which gives the frequencies for all the word tokens in your corpus per text; and a stylo_config.txt file, which lists the configurations: the dimensions of your graph, which analysis was performed, the distance measure, and much else besides. If you want to recreate a graph / analysis: here are all the parameters you need."

"The biggest problem to Petroglyph's entire "Lunch" experiment is that he did not even include a link to the raw numeric data, and just shared these visuals."
Rofl. Moving the goalposts, much? I told you exactly what settings to choose and provided you with the exact corpus.

Either way: Here is my config file for the Austen cluster. Here is the table_with_frequencies. Any spreadsheet software will be able to read those.

""Diana" actually has the "she" pronoun among its top-6 words, whereas the other two include "I" in the top-6;"
If you haven't understood yet why it is a mistake to take absolute frequencies as diagnostic features, then I don't know what to tell you.

"He is not even showing all of the text-to-text comparisons"
False! Those graphs are the result of all text-to-text comparisons. Check that table_with_frequencies! See those lines on the cluster graphs? They show the distance you'd have to cover to get from one text to the other. The longer the lines, the greater the distance / difference. Those lines are the text-to-text comparisons. Or rather, the result of each-text-to-each-text comparison.

How do you not know this?

"anybody who doubts these conclusions cannot check them to see what might have led to misattributions"
You keep repeating this. Saying it multiple times does not make it more true. I told you what software to use, under what settings, and gave you a corpus.

But alright. This is what Burrows' Delta calculates. :



Feel free to double-check by hand. Or use Excel.

>305 faktorovich:
You are fixating on these gray areas
Because a consistent application of a methodology is at least as important as having a methodology. The decision to "count" 7 similarities or not shouldn't be up to the individual case (the researcher, in their deliberations, might be biased towards the option that would be most interesting). Either count them, or not.

You're sacrificing systematic application with a clear cut-off point in favour of spur-of-the-moment decisions that might go the other way on a different day.

You're always going to "miss out" on interesting cases, either by not counting interesting 7's, or by making subjective judgments (and introducing errors). Opt for rigour and systematicity every time.

"it would be unfair to disqualify all 7 matches without considering if a given text matches other texts"
Fairness has nothing to do with it. Introducing judgments based on what would be fair makes the method subjective and open to bias. Count all sevens, or count none. Sevens could be like your "supporting top 3 phrases" -- not part of the actual analysis but interesting if other features stand out.

Apply some conditional formatting to your table: anything 10+ gets an "important" colour. Saves you from doing things by hand.

""Old-fashioned" language? Just like comedians can mimic accents and "old-fashioned" language, so can most professional writers."

...So Corelli could not have affected archaic language, but her ghost-writer could?

Answer the question: If you find affinities between Corelli's texts and those published decades earlier, is it not possible that her archaic language causes these similarities, and not some unknown ghost-writer's archaic language? If the "professional writer" can affect archaic language, so can Corelli.

"The more I apply computational-linguistic tests to texts and research these authorship mysteries, the more convinced I am that there are no coincidental matches"
I can only respond with a shrug emoji: 🤷🏻

"The 729+ possible combinations you refer to is the reason my method can distinguish between the degrees of similarity."
Sure, let's go with that. So many possible combinations are left open after you replace "percentage of passive voice" and "average sentence length" and "average word length" with a binary opposition 1/0. Let's be generous and double it: I'll grant you 1500 different possibilities.

Faktorovich has uttered insinuations and accusations of underhanded and sneaky self-serving sampling techniques in >257 faktorovich: "This is especially the case if a researcher selects chunks just as politicians in the US gerrymander congressional districts so that the final result is more elected officials that favor the gerrymandering-officials preferences vs. the average political preferences of the entire population."

Either Faktorovich genuinely does not know how sampling is done, or she is making maliciously untrue statements.

In reality, sampling is done automatically, by the software, precisely to avoid bias like this. You tell the software how many random samples to take, whether they can overlap, what the size of the sample is, etc. And the software just does it. You don't tell the software what to choose; for a 500-word sample it runs a random number generator, starts at the word of that number, and grabs the next 499 words as well. Boom, random sample.

I'll even give you a step-by-step tutorial! Call it a pre-amble to a Lunch Break Experiment (tm).

Assuming you've downloaded R and Rstudio.

1. Open RStudio
2. In the console (a ">" followed by a blinking cursor) type this or, rather, copy/paste it because it has to be exact: install.packages("stylo") and press enter. Wait for the process to finish.
3. Next, type or copy this: library(stylo) and hit enter. This will activate that package for the current session.
4. Type or paste this: CBronte_jane = readLines("https://www.gutenberg.org/files/1260/1260-0.txt")
   What this does is grab the text file at that hyperlink and save it in R under the name CBronte_jane. It's Jane Eyre.
   
5. Type or paste this: my_first_sample = make.samples(CBronte_jane, sample.size = 1000, sampling = "random.sampling", sample.overlap = 0, number.of.samples = 1, sampling.with.replacement = FALSE)
   What this does is take the text known as CBronte_jane, extract a sample of 1000 words, saving it under the name my_first_sample, using random sampling. The samples do not overlap, the number of samples taken is one.
   
6. type or paste this: my_first_sample and hit enter
7. admire your random sample (R will print only the first few lines b/c 1000 words is a lot. If you want, you can have R create a text file with that sample in it.)
8. Repeat steps 5 and 6 (either by recopying/repasting, or by pressing the up arrow until you see the command you want to run again). Every time, the sample taken will be different. That's what a random sample means!

These samples will start in the middle of sentences and the middle of paragraphs. Because they are random!

For multiple samples, just adjust the parameters in step 5 above:

1. Type or paste this: samples_10 = make.samples(CBronte_jane, sample.size = 1000, sampling = "random.sampling", sample.overlap = 0, number.of.samples = 10)
   This takes ten 1000-word samples from Jane Eyre and saves them under the name "samples_10"
2. Type or paste this: samples_10
3. Admire your 10 random samples
4. Repeat steps 1 & 2. Every time there will be a different set of random 1000-word samples
5. In the next step, you can have R write these samples to a txt file.

By adjusting the parameters of the sampling formula, you can tell Stylo to take 100 samples of 500 words, or tell it that the samples may overlap (so that, say, the last 57 words of one sample may be the first 57 of another sample). Whichever is most appropriate for the set of texts you have in front of you.

Of course, this is not limited to just a single text. You can import whole corpora into R (say, the collected works of Dickens) and run samples on all of those texts simultaneously. Not just samples, too: you can run word frequencies, emotional words, function words, ... on a whole bunch of texts simultaneously, and have the results printed in a nice table for further analysis. Just like that!

I felt like doing another Lunchtime Experiment. It's really an after-dinner experiment. But they're both meals, so similar enough to count as the same thing.

This time round, I'll look at some twenty-odd novels from the Wizard of Oz series.

I cribbed this idea from section 3.3.1 of this paper: Gladwin, Alexander A. G., Matthew J. Lavin, and Daniel M. Look. 2017. ‘Stylometry and Collaborative Authorship: Eddy, Lovecraft, and “The Loved Dead”’. Digital Scholarship in the Humanities 32 (1): 123–40. https://doi.org/10.1093/llc/fqv026.

A pdf link for those blocked by a paywall.

Gladwin et al.'s account really is taken from the following paper: Binongo, José Nilo G. 2003. ‘Who Wrote the 15th Book of Oz? An Application of Multivariate Analysis to Authorship Attribution’. CHANCE 16 (2): 9–17. https://doi.org/10.1080/09332480.2003.10554843.

So. Let's look at The Wizard of Oz series. Its original creator L. Frank Baum wrote some 14 books in the Wizard of Oz series (plus some story collections, and plays. But those don't concern us here). After his death, several people continued writing instalments for the series, most notably Ruth Plumly Thompson, who penned like 20 or so. Wikpedia says that her style was very different, more fairy-tale like.

One of the books, The Royal Book of Oz, likely authored by Thompson, was originally published as "by L. Frank Baum", but was later credited to her.

So. We have two authors with, apparently, recognizably different styles. We have one book that has been attributed to either. Let's do a rapid and basic R:Stylo run. Two questions: Will the software correctly separate all the Baum stories from the Thompson ones? And which author set will Royal be closest to? Spoiler: Binongo found that it's probably almost entirely by Thompson. But Binongo (and Gladwin et al.) do some fretting about advanced methodologies, so I, noob that I am, will just work with the defaults on Stylo and see where that takes me.

I've downloaded 11 of Baum's books off ProjGut -- I just selected the 11 most popular ones. No real triage, because this is a Lunch Break Experiment (tm). I also downloaded 9 of the Thompson books. The Royal Book of Oz I've named "Mystery_royal". Just to make things interesting, I've also dumped Pride and Prejudice in the corpus folder, along with Jane Eyre and Corelli's The Young Diana, along with Charles Stross' Accelerando, a 2005 collection of Science Fiction short stories released under Creative Commonsand downloadable for free. Click here to download this corpus as a zip file if you want to recreate / double-check my results.

I've removed all introductions, ProjGut legalese, title pages, letters to my readers, summaries of other books in the series, etc. All books start at Chapter 1.

Here is a cluster graph, run with Stylo default: a cluster graph (on the tab Statistics) and with the pronouns removed from consideration (under the tab Features).



Baum's books are in green, and they're correctly separated from all the Thompson books in grey. The Royal Book of Oz is lumped in with Thompson. So it looks like this rough and ready Stylo analysis (using Burrows' Delta) is in line with the more advanced techniques used by Binongo.

The three extra novels that are litfic/romance novels for adults are clustered together, and separate from the simple-language feel-good stories for children. As expected. The software thinks they are more like Baum's works than Thompson's. The software also thinks that Stross' science fiction is much closer to Thompson's works. Make of that what you will.

Conclusion: You really can compare any set of texts, and the tools employed will happily produce distance measures and cluster graphs and whatnot for whatever group of texts you ask it to cluster. Whether it is meaningful to do so or not.

Using the same settings, but with the noise removed:



Yeah: Thompson wrote Royal. (I also ran these tests with the pronouns left in and got pretty much the same results.)

What does this neat separation between the two authors look like on a scatter plot? (Settings: PCA (corr.) under the tab Statistics; pronouns excluded under the tab Features.)



I have nothing to add here, really. This graph is not substantially different from the previous one: they both take the same data, the same calculations as input; they're just different ways of visualizing these data.

Time to kick this up a notch! Let's try some sampling.

Instead of letting Stylo run Burrows' Delta against the entire text at once, let's extract 5 samples of 5000 words each. That produces the plot below (same settings as the last plot, but on the Sampling tab, choose random sampling, 5000 words, 5 samples, no overlapping.)



The actual text labels have become illegible, but the colours are enough: None of Baum's 5000-word samples are close enough to Thompson's (the actual dots on the plot are at the midpoint of the label name.). None of Thompson's samples have made it over to Baum's side either. all of Royal's samples are comfortably within the Thompson cloud.

Let's do the same thing, but for 10 1000-word samples per text:



Across these latter two graphs, the clouds of samples spread out a little. This is natural: some samples will contain more dialogue, or more description, or perhaps even poems and songs etc. But the take-home message is: The differences between any of the Thompson samples are smaller than the differences between any of the Thompson samples and all of the Baum samples.

It's nice to have confirmation that a fairly standard measure like Burrows' Delta gives a good enough result even without sampling. This is probably why it is a standard measure in the first place.

>310 aspirit: Nice! Your Dessert Experiment (;-D) is really fun and makes me want to play with R and Stylo. Maybe this weekend.

>306 Petroglyph: The paper you cited (https://doi.org/10.1093/llc/fqx023) does not include a list of features that is included in the Delta method, though it does mention various potential features a method could test. This lack of clarify is obviously intentional or to make the method seem far more complex with the mathematic formulas used in the description than the simple test for common-words that is obviously applied in it. This is confirmed by your statement "a word list of all the tokens in your corpus" - only word-lists are generated in the data because only word-frequency is tested.

Yes, I have seen spreadsheets before like your "table_with_frequencies". And no, "any spreadsheet software" cannot open this file, as Excel cannot open it; it only opens as a simple-text file where the rows are jumbled and difficult to interpret. This spreadsheet includes the names of the different words in the corpus and the statistical frequency of their relative appearance in different texts. The frequency of all possible words obviously creates a nonsensical data-set. What is missing in this data is how your system goes from this enormous data-set to the final seemingly simple data points on a graph - where does all this complexity go? What is the formula for selecting which of these similarities are more significant than others etc.?

You are just ignoring my points and again repeating that I just don't understand you by misinterpreting what you imagine I stated.

The graphs can be the results of fiction-writing. It is the researcher's and the software-developer's job to explain how raw data points are altered to simplify them into just a few data-points. Without this underlying specific formula of conversion, the researcher would be as well served by performing a magic trick without any science behind the transformation. There are no cluster "graphs" in any spreadsheet - it is a data-table, not a graph. There are no lines in your table; you are confusing the lines in your diagram with the data in the table. It is clear to all what your diagram is supposed to represent (the lines of similarity between texts); I am objecting that there is no rational explanation for how you arrived in this clustering-conclusion.

The formula that you do list is absolutely absurd for the following reasons. 1. By the average z-score you mean the frequency of a given word in a text. There are thousands of unique words in most texts, so how can all this data be simplified into a single average? Are you adding up all frequencies for all words and calculating the average frequency between them? This would just be nonsensical for purposes of attribution. There is no weight given to any words in this formula. To clarify your formula: k = text; author = a; n = words; z = frequency score. You are saying that by manipulating the name of the text, the author, the number of words in the text and the frequency score for all words on average, you get a single point of similarity-divergence on a graph of this texts similarity. The aren't even an author 2 or text 2 in this formula. This is the only formula this system uses to derive your attribution answers? Can you try giving an example of how this formula was applied for a specific comparison? It just seems to be nonsensical double-speak.

>307 Petroglyph: I am neither always discounting nor always counting gray-area matches, but rather using the data in the full corpus to interpret if each of these cases is indeed interesting or relevant. In the table I posted, the gray areas did not represent only cases of matches, but rather the sections of the text-to-text comparisons that on-average matched all of the texts in a given byline. You are concluding the 7-9s are uncertainties, whereas I did not include a write up in my explanation that explains them as either matches or non-matches. I do explain these variations in degrees of similarity in detail in my Re-Attribution book, as well as all of the other points we have discussed in this chat. The color-coding system is designed to guide readers of the table, so they can reach their own attribution conclusions - they can choose to count 7-9 as matches or non-matches. If I had written an article about this data, I would have dived into these questions. And all readers of this discussion can take this data and write articles of their own that disagree regarding the 7-9 matches. The point is that most of the data consistently fits into the gray regions of in-byline matches, or can be explained in the intersections/byline-crosses, and the much larger non-gray non-matches portions. You just asked me to test if my method works on the same set of texts you tested; and this data set proves that it does, as our findings matched in many of the cases; so to say that my conclusions is wrong, you would have to find fault with your own as well.

To evaluate the significance or lack thereof in archaic language that appears in Corelli's texts I would have to write several articles, starting by picking out most of the obvious examples of archaic language and searching for similar usage across British literature. Then, I would test all of the texts where similar language appears against Corelli's texts to check if there might be a shared ghostwriter between them. It is likely I would have to test several hundred texts and would have to explain the nature of archaic language in question etc. to arrive at a certain answer. But simply-put, both a professional writer and Corelli could affect archaic language; but as I mentioned previously Corelli is very likely to be a pseudonym and not the name of an authentic author given that some texts with this linguistic signature were published before she was born.

>308 Petroglyph: I followed the steps you gave for generating random samples. I see 10 first lines, some of which have garbled letters, some are just blanks without the words I would need to find these points in the text to double-check the chunks of text this system has selected. Only 1 out of the 10 starts at the beginning of a sentence, whereas the others all start in the middle of a sentence, and one includes only one word, which would also be difficult to locate in the text for double-checking. These glitches are serious problems because if the system cannot distinguish where sentences/paragraphs start and it is supposed to; then it cannot be trusted to calculate the rest of the attribution-calculations it is designed to perform. When I tried your second experiment for multiple samples, the errors of the start of samples not starting at the beginning of new sentences etc. were multiplied. But my problem is not with these types of glitches that RStudio introduces, but rather with the idea of choosing "random" samples that change in each calculation, or just choosing any samples or chunks out of a text. As I explained, the larger a text-size is the more accurate are the attribution findings derived by testing it; so it is counter-productive or an error to break any text into its pieces unless a researcher specifically needs to solve a mystery regarding who wrote the different parts of a text suspected of having 2 or more authors. Have you noticed my objections to chunking? Given my stated preferences, it would have been more logical if you gave the instructions to test the various other features you mentioned like "emotional, function words". Are these steps too numerous to list?

>311 faktorovich:
"does not include a list of features"

That's the overview article, lol. Check Burrows' own work, he explains it in detail. Don't blame me when you look in the wrong place.

This one, perhaps?. HMU if you need a pdf.

Or, you know,


This is what Burrows' Delta calculates. I can't make it more explicit than that.

"And no, "any spreadsheet software" cannot open this file, as Excel cannot open it; it only opens as a simple-text file where the rows are jumbled and difficult to interpret."

Holy shitballs, lady. Really?

Open Excel. File > Open > navigate to the folder with the txt file > double-click the txt file. (you may set an option at the bottom of the "Open" window to "all files" instead of just excel files.)

Or: right-click the .txt file. Select "Open with...". Choose Excel from a list of programmes.

Or: open a new Excel document. Drag the .txt file into the Excel window.

" What is missing in this data is how your system goes from this enormous data-set to the final seemingly simple data points on a graph - where does all this complexity go? What is the formula for selecting which of these similarities are more significant than others etc.? "

And I have answered this question several times now. I refer you again to



But if "opening a .txt file with a spreadsheet app" is beyond you, ...

You not understanding the output of a frequency table does not make the figures in that table wrong.

You keep moving the goalpostss on this: "Show me that Stylo works and how it works". I do. "Petroglyph did not show his raw data", and when I do, there's complaints that you find them hard to read. How is that my fault?

"Without this underlying specific formula of conversion, the researcher would be as well served by performing a magic trick without any science behind the transformation."

Look, if you're starting from the position that the kinds of statistics that are clearly far outside your comfort zone are inherently suspect, then there is no point in me trying to explain what they mean. You wouldn't trust me anyway -- you already don't (see your comments about elitist and gerrymandering and sneaky secretive academic cabals).

" There are thousands of unique words in most texts, so how can all this data be simplified into a single average? "
Ok, that explanation would actually require me to think pedagogically and to explain some statistics to you. That is not my job. It's not my responsibility to make sure you understand a widely-accepted and commonly used standard method in the field.

Also, you, whose method consists of manually replacing proportional measures by either/or 1/0 are wondering how a great deal of variety can be collapsed into a few general variables? Wow.

"The formula that you do list is absolutely absurd for the following reasons."
Take it up with Burrows, then. You'd better inform Maciej Eder, too (one of the main people behind Stylo).

"Can you try giving an example of how this formula was applied for a specific comparison? It just seems to be nonsensical double-speak"
Way, way, way ahead of you. I've already linked you to Gladwin et al. in >231 Petroglyph: and >309 Petroglyph:.

>309 Petroglyph: I have already demonstrated that my method works better than this standard method on a random set of texts of your choosing. There is no rational reason for me to spend the next 2 days applying my full range of tests to this particular research problem in which I have no scholarly interest. You have demonstrated in this explanation that: 1. removing pronouns confuses most of your attributions (this is why obviously the most-frequent words that tend to include pronouns should not be removed, and yet you have argued for their removal). 2. You have shown that 5,000/1,000 word samples jumble your results, but you have not stated what word-size the chunks are in your original graph. And if there are 2 different bylines in the central texts that chronologically make sense with the first author having died; there is no mystery in comparing them. It is possible your method came up with an erroneous attribution; if my method finds errors in your conclusions; you would say that my method is incorrect, instead of admitting that you have indeed made errors even when faced with detailed data in my spreadsheet that explains the error, as I have done in the "Lunch" experiment.

>313 faktorovich:

"I see 10 first lines, some of which have garbled letters,"
Yes. I told you in >308 Petroglyph: "R will print only the first few lines b/c 1000 words is a lot. If you want, you can have R create a text file with that sample in it." If you were to print all the contents of your samples, you'd lose easy and/or visual access to the rest of your code.

"some of which have garbled letters, some are just blanks without the words"
Gee. Would you agree that it is important to properly format texts before you run stats on them?

Have you made sure that character encoding was not an issue in your analyses? You do move a lot of text between various environments (txt, various webpages, html,...). But, considering we had to explain the difference between filetype and character encoding to you in this very thread, and considering that you've been doing this for how long now?

Also, why is this suddenly a deal-breaker when you don't even know if

"As I explained, the larger a text-size is the more accurate are the attribution findings derived by testing it; so it is counter-productive or an error to break any text into its pieces unless a researcher specifically needs to solve a mystery regarding who wrote the different parts of a text suspected of having 2 or more authors. "

This has been explained before: Measuring on a single long text gives you one measure. Slicing the text into smaller sections that are still of usable size (so, 3k - 10k words) gives you multiple measures for a single text. It makes your descriptions more precise!

Texts have a lot of internal variety! (To take a stupid example: Lord of the Rings. Long descriptions. Plenty of songs and poetry. Chapters that are mainly dialogue.) It's about leveraging all the evidence that you have at your disposal.

"so it is counter-productive or an error to break any text into its pieces unless a researcher specifically needs to solve a mystery regarding who wrote the different parts of a text suspected of having 2 or more authors.

What do you mean, "unless"? Texts are sliced into samples for data reasons. Authorship is only one task that corpus studies and computer linguistics can do. If a text is not by multiple authors, we've just got a more precise idea of one author's language patterns.

"Have you noticed my objections to chunking?"
I have noted them. Your objections to such a trivial operation performed on corpora of multi-million, multi-tens-of-millions of words (and sometimes individual texts) mark you as an outsider, as someone who hasn't really worked with lots of data before.

Your lack of familiarity with data hygiene (encoding, using .docx and .xlsx), as well as your not knowing how to pull a .txt file into a spreadsheet does that, too.

>314 Petroglyph: I looked at the article you cited, and features were not listed in it. The new article you linked to has a pay-wall.

For the second time, the directions you gave actually worked. I am surprised. Yes, the problem was that I had not selected "all-files". However, opening the file in Excel in a format that was not entirely jumbled did not help to convince me that this data set makes any rational sense for attribution purposes. There are 5000 different words with their frequency rates in the texts where they appear. The table is only for the "Austen" texts, and not for all of the texts. The table would have been a lot more cluttered if these rates were given for all 16 texts. Since you asked me to test 16 texts; it would be fair for you to share your full data for the full set. It remains unclear how these wildly spread out individual frequencies can be combined together in any rational mathematical way to create an attribution answer; it certainly cannot be done with the main formula you have included.

I did not mention any cabals. Gerrymandering is a very real problem that is accepted as real by academia and media alike. If you are saying that gerrymandering is equal to cabals... Then, you protest-too-much.

You are welcome to inform Burrows, Maciej etc., and they are welcome to join this discussion and explain it all to us. That would be exactly what we need to figure this out.

>316 Petroglyph: I followed your exact steps in Stylo and this generated the errors. So if there were additional steps for cleaning up the data; why would you omit those. And if once I have created an edited file, I would not be able to use it in Stylo (as I have discovered earlier); then, only online-accessible texts can be used with this software and thus they will always have these serious problems in them that will introduce attribution errors.

Unless you are claiming the songs in "Lord" are by a different author vs. descriptions vs. dialogue, for a total of over 3 authorial signatures; it would make no sense to test these components separately as genuine linguistic signatures record intuitive language usage, and not usage that changes with the genre/writing-type.

The software used for attribution should not be overwhelmed with data pieces of over 5000 words; if it is; the software designer needs to adjust this problem so that it can handle the full size of textual materials. My Renaissance corpus has 7.8 million words. So, if the size of the corpus means I am an outsider vs. insider...

All of your objections are deflecting to the central problems with your method that you have not addressed because you know they are problems that are likely to be the cause of systemic misattributions. I have proven my method words better. Yet you have not emailed me to ask for a free review copy of the 2 volumes (698 pages) in my series where I explain how this is the case in several chapters. I cite some of the articles you have mentioned in this discussion, discuss the Stylo method, and explain its faults in this book. Any researcher who refuses to consider alternative methods in a given field is doing a disservice to the progress of science.

>315 faktorovich:

1. removing pronouns confuses most of your attributions

What? All the graphs in >309 Petroglyph: are based on calculations with the pronouns left out. Look at the bottom of those graphs. That parameter is stated right there.

If I try with the pronouns included, I get this:



How does this invalidate the results in >309 Petroglyph:???

(this is why obviously the most-frequent words that tend to include pronouns should not be removed, and yet you have argued for their removal).
... their removal when you're comparing 1st-person and 3rd-person books, yes.

2. You have shown that 5,000/1,000 word samples jumble your results,
Lolwut. Not a single slice of Thompson was assigned to Baum, or vice versa. That confirms the initial results.

but you have not stated what word-size the chunks are in your original graph.
No chunks there. Full, complete, unabridged novels.

And if there are 2 different bylines in the central texts that chronologically make sense with the first author having died; there is no mystery in comparing them.
Wut? What "texts" are you talking about? Only one of the texts in that corpus has been assigned two bylines: it was originally published under Baum, later under Thompson. Because she wrote the damn thing.

"It is possible your method came up with an erroneous attribution
Sure, why not. I'm not perfect. Thing is, My results match Binongo's. Take it up with him. old sport.

instead of admitting that you have indeed made errors even when faced with detailed data in my spreadsheet that explains the error, as I have done in the "Lunch" experiment."

Well, you think my method is wrong. Few other people here seem to think so. The Lunch Break Experiments (tm) -- please get that name right, it kinda means a lot to me -- have confirmed earlier experiments run by experts. And you disagree with all of them.

Point is: I think you're wrong in calling my method "absolutely absurd". You think everyone else is wrong in calling your method, um, misguided and wrong.

>318 faktorovich:
I followed your exact steps in Stylo and this generated the errors. So if there were additional steps for cleaning up the data; why would you omit those

Look, I'll be frank. I offered those steps since they were the only way I could guarantee that you'd have access to that text. In previous posts you've expressed unhappiness at not being able to point R at your files.

But if you must: nothing prevents you from taking the cleaner CBronte_jane text that was in the corpus for the first Lunch Break Experiment (tm). Odd characters should disappear there.

Either way, the fact that an improperly encoded text causes issues is not a reason to dismiss random sampling.

Unless you are claiming the songs in "Lord" are by a different author vs. descriptions vs. dialogue, for a total of over 3 authorial signatures; it would make no sense to test these components separately as genuine linguistic signatures record intuitive language usage, and not usage that changes with the genre/writing-type.

No, what I'm saying is that poetry and dialogue and descriptions have different patterns of punctuation and content vocabulary and function words. If you slice LotR into manageable chunks, by the laws of chance some of those chunks will have more poetry in them than others, and so they will be plotted in slightly different positions than slices without poetry.

I'm not talking about "testing for authorial signatures" -- how is that the only use you can see for any of these technologies??? I'm talking about handling data and the things you can do to corpora to prepare them for analysis.

"not usage that changes with the genre/writing-type"

Uh, yes? Just one example should demonstrate this: Dialogue will have a greater concentration of punctuation marks than descriptions or more introspective passages. Several punctuation marks are part of your model, too. Including commas. Many pieces of dialogue end in a comma, before a saying verb, for instance. Also, authors will give their characters their own voices. That might confuse metrics, too. Poetry (especially renaissance poetry and 17thC poetry) will incorporate and play with older speech patterns. And the puns! And all the double meanings!

An author's signature is not 100% reliably traceable through all genres and text forms and styles. That's why you use differently-weighted parameters to test different genres.

Whether or not you agree with that last statement, you must see that the things you look at to trace authorial signature (e.g. punctuation) can change depending on character or text type, right??

>299 faktorovich: There is no bias in my re-attribution of the British Renaissance to 6-ghostwriters. I am simply reporting what the data says.

The data doesn't and can't say that the same person wrote two works. All it can say is that there's a certain degree of similarity in the style of the two works, in the ways the tests measure the style.

>301 faktorovich: "Falsifiability" is not a "key characteristic" of all science. The gravity equation g = GM/r2 might be something that cannot be proven to be wrong

Newton's theory predicts very carefully the orbits of the planet. It gets Mercury's orbit wrong by 38-42 arcseconds per tropical century.* This is a very famous test of Newtonian gravity that showed that Einsteinian gravity is more accurate.

* https://en.wikipedia.org/wiki/Tests_of_general_relativity#Perihelion_precession_...

>319 Petroglyph: The blue vs. green colors distinguish cross-byline texts in the diagram, but they exist in the same portions of the graph. And if you have defined these two bylines as distinct; you have biased the findings to favor this two-sided split.

If the first image does not include chunks and it is the most clearly distinguished out of them; this proves that the chunking method into 5000/1000 pieces only confuses the attribution results; so chunking should not be used.

I think you are repeating what I am saying about the 2 bylines, but due to exhaustion at this hour you seem to not recognize we are saying the same thing.

My method is entirely transparent and I have provided basic steps that are enough to guide a user through it. This is why you have not had to ask me to explain the steps in my method; whereas I have spent days trying to get you to offer a full list of steps involved in your method, and you have not yet been able to provide such a list. Yet again, most of the attribution results of my method matched the results of your method; so to say that my method is entirely "wrong", you have to find your method to be at least "wrong" in the attributions where our methods agree. My method is the best approach available with modern accessible computing for establishing authorial-attribution. This is a statement of fact; I have reviewed the Stylo method and most of the other previous approaches before designing my own method. I would not have designed my own method if I thought there were any working methods in the market. I have questioned you about your approach because the points I have questioned you on are those that prove the error of this approach, especially when they are explained in more and more detail as you have done in this discussion. Most articles abbreviate these explanations and their researchers are never questioned to explore the underlying problems I have focused on. I discuss these problems in the book I already published; nothing you have said has been new to me (including the Excel file glitch that I now recall I solved when I last received this type of Stylo data-set). The point of this discussion for me is to show the public the type of falsehoods computational-linguists have been claiming that have prevented attribution studies from progressing from pre-computing intuition-based attributions.

>320 Petroglyph: Because my method is more accurate, it does not fail to distinguish the authorial style even if the genre changes between poetry and non-fiction. I have explained before that groups like Verstegan's in my Renaissance corpus have drama, poetry, non-fiction etc. in them and yet they are all correctly identified. The method has to be such that genre is irrelevant. My data table for the 284 texts includes the exact comma etc. rates in the texts together with the genres these appear in. It is illogical to generalize about poetry etc. without citing specific texts where you have observed this division.

>321 prosfilaes: Yes, this is also proof that my new method of computational-linguistic attribution is the more correct phase of scientific discovery because it falsifies previous approaches, and corrects them.

>322 faktorovich:
"The blue vs. green colors distinguish cross-byline texts in the diagram, but they exist in the same portions of the graph"

What are you on about? The green book (or the five or ten green slices in the sampled graphs) is the book whose authorship we're trying to determine. The green book in its entirety falls well within Thompson's cloud. And so do all five 5k-word samples. And so do all ten 5k-word slices.

If any of these segments were majority-Baum, the software would have placed them in his half of the graph.

"the first image does not include chunks and it is the most clearly distinguished out of them. this proves that the chunking method into 5000/1000 pieces only confuses the attribution results; so chunking should not be used"

... because there's only 21 labels on that one, one per text? The next one has five samples per text, so 21 * 5 = 105 labels, and the next graph has 10 samples per text, so 21 * 10 = 210 labels. Each chunk is tested by the software and assigned an attribution, and so is plotted on the graph...

Would you like a bigger graph? Here's a 5000x3114 pixel cluster graph of 5 chunks per Oz book. I can make that bigger, if you like.

And here is a graph with 10 samples per individual book, i.e. 210 chunks that are plotted as just dots. Red for Baum, green for Thompson, blue for the text that has seen two bylines.



Either the limited legibility of every single letter in that graph is genuinely some sort of OCD thing you can't overlook, or you're just grasping at improbable straws to deny those results. Do you happen to be neurodivergent?

"I have provided basic steps that are enough to guide a user through it"
If your car won't start:
1. Take hammer
2. Hit own head with hammer
3. Repeat until car starts.

Being able to provide simple steps that others can follow is not a shorthand for your methods being correct.

"you have to find your method to be at least "wrong" in the attributions where our methods agree"

Broken clock and all that. And not all of your steps are wrong. You just look at plenty of the wrong parameters, and you conflate everything like whoa.

"Most articles abbreviate these explanations and their researchers are never questioned to explore the underlying problems I have focused on"
False. Just take a look at that Gladwin et al. article. The bulk of it is trying out various methods on training data to see which one is the best. The literature is full of criticisms of older methods, and new ones are constantly being refined.

If you complain that regular, run-of-the-mill "here's a problem, let's solve it"-type articles don't bother to explain all the basic steps of the field in language that any layperson can understand.... You don't have much exposure to scholarly literature of this type, do you?

Your comfort zone is not an appropriate measure for the correctness of a method.

"due to exhaustion at this hour "
Nah, I've had students worse than you.

I am signing off now, though. I'm going to watch another kind of noob be all cute and inept.

>315 faktorovich:
"There is no rational reason for me to spend the next 2 days applying my full range of tests to this particular research problem in which I have no scholarly interest."

Oh no, absolutely, that goes without saying. I'm not expecting you to try and replicate those results. Or whatever. I don't want you to waste your time on this, either.

>325 Petroglyph: Breaking your text into smaller chunks and making a new graph with them does not address my question. If you just answer questions with nonsense and draw more detailed pictures that represent nonsense, it is all still nonsensical. A lack of sense in an explanation means nobody can understand it; if there was sense in the logic at least somebody other than the speaker should be able to understand it.

Yes, your steps have indeed been similar to hitting your own head repeatedly with a hammer. I don't know why you have felt compelled to tell me about this process. I asked you about your computational-linguistics method.

After running several experiments, I have learned which parameters work for which period etc. The parameters I choose do not produce genre-specific errors, for example, as yours do.

Most of the articles you have cited include the same repetitive book-review section that summarizes the same basic elements of major computational-linguistic studies of the past, and then they all fail to define their own method with any specificity that is sufficient for not only reproducibility, but even for scholarly questioning of the basic premises.

The last time I played a game or watched a tutorial about a video game was a couple of decades ago. I went to the ALA convention and tried selling my graphic design skills to design board games to a group of video gamer builders recently, and they insisted that if I was not an avid gamer they would not consider me as a potential collaborator. Computational-linguistics like board game design is not an activity for the inept or for fun-seekers, and if I was inept, this conversation would not have hit 3-article-sized point. I have to translate the next 14 volumes of the Renaissance series after I finish trying to sell these 14 volumes. Digressing into attributing the rest of human knowledge is definitely a "waste of time". If this was a game; I won the experiment, and you are refusing to admit defeat by down-talking your opponent.

if I was inept, this conversation would not have hit 3-article-sized point

I simply do not know what to say. If you get everyone to tell you at length how you are failing, and trying their best to explain to you how, it's a sign that you have succeeded?

OK.

>326 faktorovich: I'm glad for you that one person thinks you won. Even if you are that one person.

>321 prosfilaes: The data doesn't and can't say that the same person wrote two works. All it can say is that there's a certain degree of similarity in the style of the two works, in the ways the tests measure the style.

This, a million times.

Such results admittedly sound rather banal in comparison with a discovery that entails rewriting literary history.

>326 faktorovich: "I went to the ALA convention and tried selling my graphic design skills to design board games to a group of video gamer builders recently, and they insisted that if I was not an avid gamer they would not consider me as a potential collaborator. "

So you've experienced rejection because of your lack of education (about games in that case) before. If gamers found your work too naive to be useful, why are you surprised that statisticians, readers, & literary scholars in this thread respond similarly? Computers are fun but by themselves they don't produce evidence. The analyst interprets data as evidence. In other words, the analyst-interpreter (you) affects the results. And, as with gaming, you're once again misled by the Dunning-Kruger effect.

Goodbye, Faktorovich. This has been such sweet sorrow. I hope you find ways to spend less of your life copying and pasting stuff in and out of web pages. May you expand your comfort zone and become a life long learner.

Peace.

(btw. You asked for the frequency table for all 16 novels in >317 faktorovich:. Here is a zip file with the table_with_frequencies, the stylo_config file, the wordlist used in the calculations, and a table with the weighting per corpus file. Enjoy!)

>331 susanbooks: The claim that the problem was that I was not a gamer was a façade to hide a rejection based on my gender, as a woman was attempting to enter the male-dominated gaming industry. They were not asking me about my knowledge of gaming, but rather of my practice of gaming; a designer has to know how to design, not waste their time gaming instead. "Computers are fun" - you are just saying random unrelated stuff. Computers "by themselves... don't produce evidence"? You mean like a computer that has been turned off - producing evidence on its own AI power? I know computers don't produce evidence themselves. That's why I invented a method where I use a computer to create the evidence. Before data can be interpreted, the correct data parameters have to be collected in a rational and systematic manner that solves the problem and leaves evidence behind of what the collection process was for others to check it for errors. My process is thus correct, while simply plugging the data into the Stylo software despite the various glitches this program has is incorrect. There was no bias (Dunning-Kruger effect) either in my attempt to design board games (as I had a successful track-record of designing over 300 books that qualified me, and the only bias was the gender-bias on other side), nor in my logical interpretation of the data I have gathered about authorship in the British Renaissance.

>332 Petroglyph: At the top of your "wordlist" file it says: "# This file contains the words that were used for building the table # of frequencies. It can be also used for further tasks, and for this # purpose it can be manually revised, edited, deleted, culled, etc. # You can either delete unwanted words, or mark them with "#"" This is the definition of bias, the 5,000 words your "Lunch" experiment tested for was not a randomly-selected list, or a list of the top most-frequent words in all of the texts in the corpus, but rather a list you manipulated with "edits, deletions" etc., as you explained in your comments about pronouns. You check the results of a random test before deliberately removing the words that lead your conclusions to anything other than a 95%+ re-affirmation of the existing bylines. This how your outputs are skewed to appear to show correct results. If you method was fair, rational and scientific, there would not be a step where the wordlist could be thus unfairly adjusted.

>333 faktorovich:
"not waste their time gaming"
I wonder why they didn't let you design things you clearly think are beneath you. Must be their prejudice.

>334 faktorovich:
"a list you manipulated with "edits, deletions" etc., as you explained in your comments about pronouns"

How much time do you think I spent on those things? Do you think I have the time to trawl through 5000 words and check every one?

Read what that file says again, but more slowly. If your text contains words you don't want the software to count in its frequency tables, you can put a "#" in front of it, and on the next run-through the software will skip all the words you've marked as irrelevant. If you've pulled your texts from ProjGut, you may want the software to skip words such as "Gutenberg", for instance, or "tm". Another use-case is that, for plain-text versions, the place where html and epub versions have illustrations often just say *Illustration* -- those clearly aren't something you want to take into account. Now, you can either a) write a little program that deletes all of the occurrences of *illustration*; b) you can scroll through all of the texts in your corpus and delete them manually, or c) you can tell the software to ignore the string *Illustration* altogether by hashtagging or deleting it in one central location.

It says you can. Not that I have done this. Very important distinction, that. The distinction between truth and deceit.

I'm getting tired of your unfounded accusations of me manipulating data. I don't do that.

I will now prove to you that you are making false accusations.

If I told the software to ignore words by hashtagging them, you should be able to find those. Go on. Check: open that file with notepad, press ctrl+f and search for #. The only place you'll find # is in that notice you copied in your message.

I did not delete any words, either. If I did, there'd be fewer than 5000. Copy-paste that word list into a spreadsheet and see how many rows there are. You'll find there are 5000. No deleted words.

Go on. Check. Then come back and apologize for your baseless and malicious accusations of me deliberately skewing my results in order to make you look bad.

A question that keeps getting ignored despite a few people hinting at it so I will just ask.

A lot of the tests in this methodology are quantitative - down to punctuation, spelling of words (comparison of usage does not account for various spellings as far as I can say), abbreviations and other similar elements which are as much a feature of an author style as they are of editors', printers and publishers' house rules. And with the texts of the British Renaissance that's even more pronounced because of the fluidity of both punctuation and spelling.

All tests seems to have been ran on the texts from the printed corpus. Which is fine as long as the deviations are taken into consideration before looking for similarities. Not all manuscripts are available (unfortunately) but there are enough of them in various libraries and institutions which can be made available to scholars (or are sometimes available on microfilm or even in a legible transcribed "as is" copy). How many of those manuscripts and from how many different authors (as per the current attribution) had been inspected and compared with the texts available in the corpus to find out the differences in these elements between the two texts - thus allowing either these differences to be discounted if minimal or to be taken into consideration when comparing the texts? Or are there any studies that had done that comparison before where possible?

>333 faktorovich:

I am going to call bullshit to that somewhat. Why should a video-game company pick someone who knows zilch about gaming to design a boardgame, who hasn't had any experience of designing boardgames, who doesn't play boardgames or video games, rather than go to one of the hundreds of experienced companies and individuals out there who know both video-gaming and what works in a boardgame design today. Boardgames today are not, or at least the good ones are not, really like most of the mass market boardgames that people think of.

But of course you were not talking about game design you were talking about graphic design - there is absolutely thousands of commercial artists and graphic-designers out there, all with portfolios, many with experience of practical design in the field of board games. Without a history of previous work or a portfolio of work* no-one is going to employ you in that field. This is pretty normal in many fields of work.

* I have seen some people get work through a portfolio of their redesigns of existing boardgames. But that redesign work has been public and widely praised.

I don't see how you could do design work on games if you don't game yourself. It's one of those areas where unless you have been there yourself, you will not see the problems. It's no good having a game that looks beautiful if it's not playable.

>336 AnnieMod: For most works it is rare for there to be a single manuscript (or typescript). In the 20thC examples in my field the norm was to compose on a typewriter and then there would be hand corrections, usually by the editors paying for the story (book packager like the Stratemeyer Syndicate or publisher).

On some occasions, after a marked-up draft was returned to the author, portions or all of the typescript would be returned to the author for rework.

I have seen and we have some examples where two or more drafts of a typescript are extant. The initial one can have hundreds of corrections marks in a couple hundred pages. This becomes a lot of work for the editors and may be a reason not to ask a given writer to work on later volumes in a pseudonymous series.

Here is an example from our collection for Trixie Belden #22 Mead's Mountain. It is from the mid-1970s. In this case there are colored pencil marks for each editor to made notes on the typescript.

In some cases the edited manuscripts have large insertions by an editor that was intended to replace something that was not desired from the author. This kind of change would tend to skew a bona fide authorship analysis. Yet, you only know it occurred if the manuscript with corrections is at hand.



This is a fairly easy example since the first draft was typed. When the manuscript is holograph (handwritten) then things get much harder since there is the issue of chirography (reading the handwriting). Some authors are notoriously bad about punctuation and this was often left to the publisher and sometimes generations of publishers who engage in copy editing to make the book conform to the publisher's style. For these and other reasons, I am concerned about relying too heavily on details like punctuation (exclamation points, etc.) as a marker of an author's style. Often it comes down to what the publisher will allow.

James

>339 Keeline:
>335 Petroglyph:
>321 prosfilaes:
>285 lorax:
and others
It's been a real joy to have voices in here that know what they are talking about, and are able to write about it clearly.

>339 Keeline: We are talking about the 16th century here - not about the 20th though.

I completely agree that 20th century manuscripts are a question of interpretation (and there many versions around depending on what you count). There are sometimes more than one for the 16th century books as well but nowhere near as many (for obvious reasons) and even when there are more than one, my point still stands - if that manuscript's quantitative analysis is materially different from the published text and the published text is used for author attribution, is the attribution for the writer or for the editor/printer? Add to that the lack of rules in both spelling and punctuation and those differences may multiply just because whoever sets the book has a different idea of spelling or using punctuation.

Which is why I asked how many texts had been verified against the existing manuscripts or if such a study had been done before and if so by whom.

What >340 MarthaJeanne: said!

>340 MarthaJeanne:

The way this topic eventually went, and the people who brought it there, is everything that's good about the LibraryThing community.

>341 AnnieMod: I think that the problems of older manuscripts multiply the problems of simple examples like the one I showed. Reading just letters from some people can be quite a challenge, let alone full book manuscripts in holograph.

There are cases where the typesetter/publisher have misread a manuscript and put the wrong word in. Sometimes this is caught at the galley stage. Other times it is only caught and corrected after publication. When an error is corrected early in a book's publication history, it can become a "point of issue" — a way to distinguish between copies with the error and ones where it has been corrected.

I think that most textual analysis is done on the published versions, after the editors have had their input. Sometimes it is generations of editors. Thus it is hard to get as close to the author as one might like.

Even when there are manuscripts extant and available, getting these into machine-readable text requires a lot of work and it is probably only rarely ever done because of this.

There are no manuscripts for Shakespeare's plays, for example. The Quartos, many of which were published during his lifetime, are probably all unauthorized and thought to be the result of someone with a good memory in the audience writing down the lines or a person acting one role and doing his best to remember the rest. Most of what is called Shakespeare comes from the four folio editions. The first of these was published years after his death in 1616 in 1623 by his friends. Even the selection of plays included varies a bit from edition to edition. Some plays are only known from their inclusion in the folio editions.

Since it was not possible to have the type remain "standing" for the life of one or more editions, it was fairly common to reset the type for each new edition. This can account for variations in spelling and typesetting from one to the next. Technologies of reproducing page plates like stereotype and electrotype were of the 19th Century.

If you go to an online text, which will you get? My wife has a Ph.D. in English with a specialty of Shakespeare so my limited knowledge is based on things she has said over the years. Lately she directs open readings of plays. A scene is cast on the spot (in person or via Zoom lately) and people read from the versions they have. But it is very obvious that the available printed or online copies vary a lot, including to whom lines are attributed and whether sections of scenes are included at all. This is why the performers in a production normally have to all work from the same copy.

What effect would this variation have on textual analysis. The immediate answer is quite a bit.

We might say that the First Folio of 1623, if carefully transcribed, is the gold standard but this is after William Shakespeare of Stratford was dead and buried so he was not around to provide any edits. The plays in his lifetime probably were adjusted over time when reperformed. Some look to plays like Macbeth and wonder if some scenes were added at a later date and quite possibly not by Shakespeare.

How many 1500s and 1600s literary and dramatic manuscripts are available to consult and analyze? Probably there are very few. So it is probably a safe bet that this is not part of the usual or new textual analysis because it is so hard to get the material prepared for that analysis.

But I should stick to the 20thC series books which has been my area of specialty for 33 years as a collector, bookseller, researcher, and writer of popular and academic articles and presentations. It's not my day job but it is my long-time avocation.

James

>344 Keeline: And if a study which attempts to prove that a huge amount of works had been mis-attributed does not do the work of looking at the available manuscripts, what kind of study would do that? The whole thesis here is that everyone had been wrong for a very long time after all. Proving that any similarities based on some of the elements are actually meaningful is not a useless exercise even if it is not trivial by far.

If the answer is that all the work is done on the printed versions available for machine analysis, that's fine. If the answer is that none of these texts (or texts by the same authors) actually have an available manuscript, then that solves that question.

Shakespeare is... Shakespeare. But his troubled publishing history is one of the reasons why I am wondering how definitive is the comparison of punctuation and abbreviations and so on in printed texts of the period and how much they show an author's style really... :)

>335 Petroglyph: You previously said several times that you delete pronouns from your tested words, now you are denying "trawling" through the words to screen out words you view are inferior for attribution? So then the words you choose are the most frequent 5000 words in the texts? Such a large top-words list will include most of the words in any given text given the high frequency of the most common words. Comparing such a large quantity of words against each other would create a 5000X5000X#of-texts dimensional mathematical space unless there is a trick that simplifies all this or evaluates something simple - or your formula that obviously does not fit your description of your methodology. You are just repeating that you do delete some words with the # mark as "irrelevant", while also insisting that you do not "trawl" through them to delete them. Why would you keep "Project Gutenberg" or "Illustration" in your text-file. I always delete these elements during the glitches-editing stage when I clean up files to prepare them for testing. If you fail to delete these elements, your word-total for the text is wrong, among other things. My accusations against you skewing results are based not only on my previous research of data manipulation in most published computational-linguistic studies, but also in the various contradictory and false remarks you have made across this discussion that I have pointed out. I have not referred to general falsehoods in your case, but to specific instances. You have ignored most of my criticisms, except for those that you think you can explain. Your failure to even explain the formula you said you are using is just one example of how you are not making a full disclosure here of your method, and if you did, even more falsehoods would become visible.

>339 Keeline: My conclusion about the British Renaissance is that a Ghostwriting Workshop of six worked collaboratively to create it; a portion of the texts have a very strong single authorial signature, while another portion have two or more authorial signatures. My data tables show the degree of collaboration and between whom collaborations took place in each text. As I mentioned earlier, at least three of these ghostwriters were publishers (Verstegan, Byrd and Jonson), and as part of these roles they would help prepare texts for publication by not only typesetting or organizing the typesetting process, but also editing handwritten manuscripts for print. This is why there are occasional very minor tertiary signatures in some of these texts, where the collaborator played an editorial role, and did not write a significant portion of the full text. The size of my corpus (284 texts) and the 27 different tests applied to it makes it easy to pinpoint even these minor editorial contributions. Only if changes are too minor for tests to register them would editorial input not be noticed by my method, and in those cases the collaboration effort is too minor for an attribution.

>343 timspalding: "The way this topic eventually went, and the people who brought it there, is everything that's good about the LibraryThing community."

Yes! If nothing else, this thread has demonstrated how many smart, patient, articulate people participate in these forums. The fact that they all list libraries I can browse for book recs is wonderful. Huzzah LibraryThing!

>343 timspalding: Couldn't agree more. And I love that LT supports such a thread rather than shutting it down and castigating members for daring to rebut the work of an author.

Petroglyph, let us know if you ever decide to publish your Lunch Break Experiment (tm)! It could take off like the For Dummies series!

>344 Keeline: I explain that there are "Shakespeare" (i.e. Percy) play manuscripts or those that match "Shakespeare's" handwriting that have been either pointed out by other scholars or by me in my Re-Attribution Volumes 1-2. Actors only learn their own lines, and not the entire play. And troupes were staging new plays in as few as a week or a few days (according to "Henslowe's Diary"), and it would be impossible for any human to memorize a new play (17,000-30,000 words) in a few days; you must be imagining the modern play production process where a troupe spends a year or more repeating perhaps the same re-running play, or at least gets to rehearse for months before the first staging; after all that repetition, a play might be mostly remembered, but try sitting down for 2 days and remembering as much of any play as you can, and you will have an estimate of how unlikely this is. Around half of the "Shakespeare" plays were published decades before the first folio added over a dozen plays in 1623. I explain the complexities of this etc. across the series. The heavy edits between "Hamlet's" first 1603 and second 1604 quarto is explained in my translation of this previously untranslated first version of this play.

Typos in a transcription that misplace a line from one speaker to another have absolutely no impact on my attribution tests, as the different character names would register as equivalent nouns in the linguistic elements my method counts.

Yes the fact that around half of "Shakespeare's" plays were first-published years after his death is one of the reasons for my re-attribution of these later plays to mostly Percy and Jonson as the underlying ghostwriters behind this pseudonym. Percy and Jonson collaborated in various degrees in their playwriting, and this is why previous computational-linguistic/linguistic studies of these plays have found signs of more than one author contributing. Obviously, most of the 1623 plays were either written close to that date or were completely or very heavily re-written after "Shakespeare's" death. There are several chapters in Volumes 1-2 and in the translations (Volumes 3-14) where I explain this history, past misattributions etc. fully.

EEBO has hundreds of texts from this period available in an accessible digital format for testing.

>350 faktorovich:

you must be imagining the modern play production process where a troupe spends a year or more repeating perhaps the same re-running play, or at least gets to rehearse for months before the first staging;

You clearly know very little about "the modern play production process". With the exception of long-running Broadway-type hits, most plays run for a couple of months, if that. And the cast rarely, if ever, gets "months" to rehearse. A few weeks, including tech, at most.

>346 faktorovich:
"You previously said several times that you delete pronouns from your tested words"

Remove from consideration. Not delete from the corpus. Just tell the software not to take them into account.

1. Whenever I do so, I am explicit about it.
2. The graphs produced by the software explicitly note at the bottom whether pronouns have been removed.
3. I have been open about my reasons for why I (and numerous other scholars) do so: this removes some of the artificial distance between first-person novels and third-person novels, caused by their differences in pronoun proportions.
4. The only way you even know about the "don't include pronouns in this graph" is because of #1 and #2.

I would say that this openness and pro-active reporting of the steps taken in generating the graphs are kinda the opposite of the kind of sneaky and underhanded manipulation you are accusing me of. You still owe me an apology for that, doctor Faktorovich.

"now you are denying "trawling" through the words to screen out words you view are inferior for attribution?"

• Don't put words into my mouth. I don't consider pronouns to be "inferior for attribution". They're very useful. But I do recognize that, sometimes, in some genres, they have an outsize impact, and it can be useful to look at graphs with and graphs without them and compare them (Can your method even do that? Within seconds? Of course it can't!).
• Uh, yes? I don't scroll manually through word lists, grinning evilly as I delete words *by hand* left and right. Here is the total extent of what I do to remove pronouns:

It's a standard option. Right there. See that orange circle? That points out a square you can tick. As a standard option. I cannot stress this enough. In a standard piece of software, used every day, by experts. (And sometimes in n00bish Lunch Break Experiments (tm) on LT). If you have issues with this that the people who code this and who use this every day haven't noticed, well, either you're a lone genius of brain-melting insight, doctor Faktorovich, or you don't know what you are talking about. Given your skills with .txt files and Excel, I'm going to side with the professional Computational Linguists on this one.

Your mindset is that of someone who scrolls a lot and does things manually. You're assuming wrong in thinking that the professionals think like that.

You seem to be thinking that it's either "all the pronouns, every time, every test, not a single exception" or "This person must think that pronouns are never useful."

Neither is true. Stop projecting your black-and-white thinking onto me. I don't think like that.

"would create a 5000X5000X#of-texts dimensional mathematical space"

How can you ask me multiple times for the exact configuration of my data, and then proceed to skip the relevant details? You invent this ridiculous caricature of the method even though the config file is right in that zip file that you downloaded. The word list merely lists the 5000 most common words in the corpus used for that analysis. Nowhere does it say that it's a "5000X5000X#of-texts dimensional mathematical space". That ridiculous conclusion is all yours. That is a story you've concocted in your head and then, without even doublechecking it against that config file that you have at your disposal, you come into this thread and pretend that your misunderstandings are mine.

Use the sources at your disposal, doctor Faktorovich. You should have learnt how to do that as an undergrad.

"You are just repeating that you do delete some words with the # mark as "irrelevant", while also insisting that you do not "trawl" through them to delete them. "

Again this stupid accusation. Go search the word list file. I dare you to find a single word that I have hashed out. I haven't, and your repeating this accusation is a lie.

I listed reasons for why someone *might* exclude some words from consideration like that. I also stated very explicitly that I did not do so.

You owe me a retraction and an apology, doktor Faktorovich.

"Why would you keep "Project Gutenberg" or "Illustration" in your text-file. "

Your small-scale, copy-paste intensive, one-text-at-a-time workflow is showing again. It is entirely possible to write a piece of code that crawls entire websites (such as ProjGut, or newspapers, or online archives such as the Proceedings of the Old Bailey) and that retrieves all the texts that match its requirements (published in period X; between lenghts x and y; of genres A, B or C.). Thousands of texts for analysis can be harvested like that. Tens of millions of words. Hundreds of millions of words. At that point, it's good to a) have a look at word lists to see if any distorting crap has made it through the harvesting and automated cleaning and prepping, and b) have a central place to nix it.

Your plodding and time-intensive copy-pasting works for you. I'm sure you've invested lots of time in it. But computational linguists don't work like this. Not at all.

"Why would you keep "Project Gutenberg" or "Illustration" in your text-file. I always delete these elements during the glitches-editing stage when I clean up files to prepare them for testing."

Oh. So when you do it, it's prepping your texts. When I do it, out come the accusations of manipulating the data. Interesting.

"If you fail to delete these elements, your word-total for the text is wrong, among other things."

Let us, hypothetically, consider the case of a 5000-word list with a single hashtagged-out word. Question: does the software then only consider 4999 words? Or does it bring in the 5001st word, in order to keep the "most frequent words" number constant at 5000?

I will point out that any statements you have made concerning hashtagging absolutely rely on you knowing the answer to this question. So you must know.

"My accusations against you skewing results are based not only on my previous research of data manipulation in most published computational-linguistic studies"

Prior to this thread you had not even heard of me. You cannot have included me in your previous research. Therefore, it is impossible for you to have demonstrated any results-skewing on my part in the research performed at a time when you were unaware of my existence.

This is a pretty poor example of your reasoning skills, doktor Faktorovich.

"the various contradictory and false remarks you have made across this discussion that I have pointed out"
You have, on numerous occasions, misunderstood even the simplest things (involving bollocks, ducks, attics, character encodings, ...). You conflate "I am right" with "Others are wrong". You conflate "casting aspersions on not-my-model" with "yes-my-model is correct". You conflate "Doktor Faktorovich thinks this is nonsensical" with "this is nonsensical". You engage in a lot of black-and-white thinking, and are very quick to cast yourself as the victim.

Such, I'm afraid, is the lot of all Revolutionary Mavericks. Take courage in the though that no prophet is accepted in their own hometown. You are not the first Maverick to suffer such pangs and such slings and arrows. You won't be the last.

"You have ignored most of my criticisms, except for those that you think you can explain. "
Gish Gallop.

"Your failure to even explain the formula you said you are using is just one example of how you are not making a full disclosure here of your method, and if you did, even more falsehoods would become visible."

More sloppy thinking, Doctor Faktorovich! You seem to have confused me with someone who owes you an explanation. You are not entitled to my time. You are not entitled to my teachings. Fixing the gaps in your knowledge is not something that I am responsible for. I am not your daddy, your mummy, your teacher, your mentor, your colleague, your student, your editor, your pet-sitter, your sassy gay friend, *or* your weed dealer.

There's tons of stats help out there. Avail yourself of it. Whining that people who disagree with you on this stuff aren't taking the trouble to undertake unpaid working hours to teach you is entitled and unbecoming of someone who includes "Doctor" in their byline.

>334 faktorovich:
"You check the results of a random test before deliberately removing the words that lead your conclusions to anything other than a 95%+ re-affirmation of the existing bylines."

Here is where you accuse me yet again of doctoring my results in secret without letting you know.

Apologize for your lies about me, "doctor" Faktorovich. Show me that you can be a person I respect.

>336 AnnieMod: All tests seems to have been ran on the texts from the printed corpus

AFAIK plenty of the kind of text editions that are freely available online (e.g. StandardEbooks, or Sacred Textsin addition to the usual suspects) just silently modernize the punctuation and the spelling.

For instance (and I am absolutely not implying that Dr A. F. used this particular website), this section of the very extensive StandardEbooks stylesheet explains that they normalize things like italicization -- including for emphasis -- and punctuation.

>353 Petroglyph: Yep. The Bibliography lists the sources - most are Early English Books Online, some are Project Gutenberg, Oxford Text Archive and other available online archives (plus some modern editions). That's what triggered the question to some extent - without accounting for the level of changes first the publishing process back in the day and then the digitization had caused, some of the metrics can be very skewed.

There are tests that can be ran on a corpus with these characteristics of course - but punctuation comparison used to note similarities in style just does not sound like something that would work. A few people brought it up earlier in the thread but it was mostly ignored so I thought I'd ask directly.

>352 Petroglyph: There is no difference for the linguistic conclusion if you delete a word or tell the system to ignore it in the computation.

Pronouns are some of the best indicators of style that very rarely are influenced by the first vs. third person, but even if they are influenced by this element, then this preference to write in first vs. third person is itself one of many characteristics of a given author's linguistic-style.

I prove that the standard Stylo attribution method is underhanded and erroneous in detail not only in my specific replies (most of which you have ignored), but also in the forthcoming Journal of Information Ethics article. Since I have supported every statement I have made in this discussion with evidence, I have proven all statements I have made and thus I stand by all of them as-is. Across this discussion, I hope I have made it clear that I am objecting to the standard Stylo and other very similar methods, and not to you personally, or to your application of this method. The steps you have described match the standard steps described in other computational-linguistic articles. The problem is not your application of this Stylo method, but the errors in the method itself. You began by claiming that the Stylo method tests for hundreds of different features, but now we are clearly discussing the application of the single word-frequency test that considers the 5,000 most frequent words in a corpus. If you try to build a table of potential combinations of the top-5,000 words in any text you will run into the problem of having to randomly exclude most single-occurrence words while keeping others. Are the words that appear in other texts favored above those that only appear in 1 of the texts or the other way around. If you try to create a drawing of 5,000 words in any text and 5,000 words in another text and draw lines that show the similarities between them; you are going to have a chaotic set of intersections vs. non-matches. Measuring the comparative frequency of every one of these 5,000 words that are most frequent in a group of texts is like coming up with 5,000 different relative features of a group of humans - comparing the nose-length, the eye-color, and tiny differences in elbow-length and 5,000 other such miniscule elements; then you are saying this system comes up with an average of all these elements that is a single number that measures how similar or different each of these humans are to each other; but what if they happen to have many tiny differences but they are basically the same overall height etc.? This chaos is multiplied if a corpus grows to 6 texts (as in your Austen example), and is so complex that you did not even attempt to send the full statistics file for the 16 texts in your "Lunch" experiment. If these words were DNA, 99.9% of them would be identical in all samples. But in literature, there are many words that are present in some measure in most texts. My test for the top-6 most-common words can be easily represented in a table for researchers to question what these frequencies mean in individual texts; what words are uniquely frequent in a text is the element that registers that major intuitive tendencies of the author. The rest of the vocabulary depends heavily on the narrative/ setting and various other elements that are measures of story-structure and not the linguistic-signature. All of the computational-linguistic articles I have read from the past couple of decades use this same approximate 5,000-top-words or some other unstated number of top-words to measure similarity, but their introductions claim that their method is uniquely complex and sophisticating. I have shown how Stylo introduces typos into texts when it breaks a text into pieces; even the introduction of a single mistake that is not in the original text means a given method should not be used as the program has un-repaired problems that can when multiplied across a text lead to a misattribution. You are repeating that using Stylo takes only moments or a lunch-break to perform an experiment, but expedience is in this case at the cost of scientific accuracy. Computational-linguists are making the process of simply opening a folder and running the standard check sound like complex programming. In truth, my method multiplies the accuracy of Stylo by at least 26 times as Stylo's 1 test is turned into my 27 tests. You have not presented evidence of Stylo or any other program being able to run the same 27 different tests my method performs. And the basic steps you have attempted to show have glitches that prevent a researcher from applying them to test the results this system generates on a larger corpus, etc. Your diagram shows that Stylo only has an option for testing "words", not punctuation and other features you advertised it can test.

Your obsession with my use of copying and pasting is mesmerizing. I could create a program that would execute the steps of my 27-test method and then it would be a software that would pop-out the answer instantly with a nice table like the one I included. But as a woman without any connections; I am not going to get any funding to develop this program or to market it to researchers. And even if it was the most brilliant program in the market; researchers like you would consider my gender a disqualifier and would refuse to use it even if it was free. The point of an authorial-attribution method is that it has to lead to an accurate and defendable attribution claim; and my method delivers on both of these points as I have demonstrated. Just because you are using software and I am automating the steps in already completed software programs that require a few extra steps to process the data; this means that my method requires more input labor; but labor is not a fault, but instead just a part of the scientific process.

>355 faktorovich:
"If you try to build a table of potential combinations of the top-5,000 words in any text ..."
To reiterate: this is how you imagine that Stylo analyses things. To reiterate: go check that config file. To reiterate: you have a PhD; you should know to check your data. Especially if the people you criticize have told you twice now that your misinterpretation can be corrected by actually looking at the documents you asked me to give you.

"Since I have supported every statement I have made in this discussion with evidence, I have proven all statements I have made and thus I stand by all of them as-is."

Yes, that is how things work in LitCrit: if you can support your reading with some arguments from the text you're good. But we're talking about maths here. Your arguments actually have to be objectively, mathematically correct here.

Your PhD in English and LitCrit makes you an expert in zero other disciplines.

(I will also note that you have no idea what degrees I have, or whether I, indeed, have any. Reflect on that before you say things about my credibility).

"This chaos is multiplied if a corpus grows to 6 texts (as in your Austen example)"

You keep going further and further down this "5000 x 5000 x # of texts" path, which I have repeatedly pointed out is your misinterpretation of the case. Not mine. CHECK THAT CONFIG FILE.

Do you, perhaps, require assistance in understanding it? No shame if you do. Just tell me.

"you did not even attempt to send the full statistics file for the 16 texts in your "Lunch" experiment. "

You lie, Anna. You are a lying liar who lies. >332 Petroglyph:
And I know you have seen that message and downloaded the zip file because of >334 faktorovich:

"I have shown how Stylo introduces typos into texts when it breaks a text into pieces"

Such amateurism. Like an impatient toddler. You have done no such thing. You were confronted with a piece of software importing a text from the internet that was inappropriately formatted. Instead of thinking, "Huh, I should replace this version with something that is in UTF-8", you just, incredibly, give up and blame the software. I also said in >320 Petroglyph: "nothing prevents you from taking the cleaner CBronte_jane text that was in the corpus for the first Lunch Break Experiment (tm). Odd characters should disappear there." So there was your solution. But no. Just throw your hands up, walk away and grumble about how if it's too difficult for you on the first try, it must suck completely, in general and in specific.

At the first glitch you encounter, you try nothing, and you're all out of ideas. And then you blame the software.

What do you do when your method introduces odd characters and mis-handles carriage returns? Excoriate your method and give up? Of course not. You fix the problem in the text and move on.

Double standards, "doctor" Faktorovich.

"even the introduction of a single mistake that is not in the original text means a given method should not be used as the program has un-repaired problems that can when multiplied across a text lead to a misattribution"

Wow. If a single mistake can throw your method off, then maybe we're better off not trusting it at all. There's like no tolerance for erroneous characters, no robustness at all.

Are you completely sure there isn't a single page number left in your Gutenberg files? There's a drop cap in those Percy sonnets. What about the curly brackets that MS Word introduces? Have you taken any measures to correct them? Like, since >205 melannen: and >219 Petroglyph: and >276 Keeline: explained character encoding to you. You weren't aware of that issue when you did all the analyses you wrote your self-published books about. Are you sure not a single erroneous character isn't in there throwing your results all out of whack?

If you like, you can give me your corpus. I can run some Computational Linguistic tools on them that'll identify unorthodox characters.

"Your obsession with my use of copying and pasting is mesmerizing"

Because it is so cute and quaint and precious! Like watching somebody type an entire book with two fingers! I knew a guy once who was writing his MA thesis in public-private key encryption, and the way he typed capital letters was to hit the CAPS LOCK key, find the letter he wanted by hovering his index finger over the keyboard row by row, hit it, then hit CAPS LOCK again and continue typing with two fingers. It was baffling and adorable in equal measure.

"Computational-linguists are making the process of simply opening a folder and running the standard check sound like complex programming"

Look, I hate to break it to you again, but your comfort zone is like, your issue to deal with, you know? It's not the measuring stick by which to judge statistical methods employed in applied mathematics and computational linguistics and developmental biologies and a whole bunch of other fields that you do not have nearly enough experience in. Your massively un-streamlined tinkering (not even conditional formatting in your table, like, really??) is so far behind the standard tools of the trade that you don't seem to realize just how far behind you are. Or perhaps you do, and that's the source of so much of your hostility. But I will refrain from armchair-analyzing you.

"I could create a program that would execute the steps of my 27-test method and then it would be a software that would pop-out the answer instantly with a nice table like the one I included"

Our conceptions of "nice table" differ. But that's subjective. But yeah. Excel functions and macros are a thing, you know? And they can get you part of the way. Look it up. Acquire a new skill. It'll do you good.

"But as a woman without any connections; I am not going to get any funding to develop this program or to market it to researchers."

Ah, I can't speak to how women are treated in computational linguistics. If you frequent those circles, you'll know better than I do...

"researchers like you would consider my gender a disqualifier"

... but I draw the line at even more ugly smears about me. I know why you do it, but it's shit-flinging all the same. I do wish you wouldn't assume things about me.

"would refuse to use it even if it was free"

1) your method is already free.
2) we refuse to use it because it is based on naive misconceptions and because it completely obliterates any relative values in favour of 1/0. (and a whole 'nother bunch of reasons)
3) I will say that all of the software that we've talked about in this thread, R and Rstudio and Stylo and antconc are absolutely free. MS Word and MS Excel are not. Don't give your money to Microsoft. Support open source software instead!

"You have not presented evidence of Stylo or any other program being able to run the same 27 different tests my method performs."

1. Oh, but it can. Have a look at this reference manual in pdf. For things like "passive" you have to, of course, run your corpus through a morphosyntactic tagger first, and then write your own function to extract them. So those are a bit trickier.
2. Once again, it is not my job to explain everything about stylo to you until you are willing to accept my explanation. Look it up, doctor Faktorovich.

"In truth, my method multiplies the accuracy of Stylo by at least 26 times as Stylo's 1 test is turned into my 27 tests."
... I'm not sure I follow. Could someone who's more mathy than me explain? My feeling is that, since there's two possibilities here (either Faktorovich is right, or I am), so there's only a 50% chance of that being true. 🙄 I dunno, though.

"The point of an authorial-attribution method is that it has to lead to an accurate and defendable attribution claim"

Here's your LitCrittiness showing up again. It is not enough for a claim to be minimally "defendable". You're trying to base your method on stats here! Those do have a right-or-wrong answer if you misapply them!

"and my method delivers on both of these points as I have demonstrated"

You've demonstrated it to your own satisfaction only. I'll say this again: either you're a lone genius of brain-melting insight, doctor Faktorovich, or you don't know what you are talking about.

You still owe me that apology, Anna Faktorovich the liar. And the longer I have to wait for it, the more opportunities I'll have to call you a liar. Because you tell lies about me.

Awesome. *munches popcorn*

>357 MrAndrew:
I'm nibbling on crisps and playing some bangin' tunes in the background. We must be up to, like four and a half article by now. Excellent!

>356 Petroglyph: You argument has now just deteriorated into completely nonsensical yelling as you are attempting to attack my credibility, even as you repeated that I have a PhD. I did not check if your profile includes your name or credentials. It is a good point that you might have no credentials at all and this might be the first time you are reading about computational-linguistics, as you are suggesting in your comments. I tested every step, concept and looked up every paper you have brought up in this argument; I have explained my objections to all of them already; these objections are weighty and prove the method you are using to be faulty. There is nothing new you are saying in your latest tirade. My directions refer to a "spreadsheet"; they do not specify that users have to use Excel; they can use any free spreadsheet software. You are saying that I would have to write a program that includes my 27-tests to adjust Stylo to my method. A better approach is for me to write a new program outside of Stylo for the 27-tests to avoid the glitches that Stylo inserts to corrupt texts it processes, as I explained earlier. All of the statements I have made in this discussion were the exact truth; the louder you deny them, the more it sounds like you keep protesting-too-much against having your method questioned, instead of presenting evidence to the contrary.

>359 faktorovich:
Sure, whatever.

I'm afraid, though, that I cannot accept this as an apology for your repeated lies about me.

>355 faktorovich:
"In truth, my method multiplies the accuracy of Stylo by at least 26 times as Stylo's 1 test is turned into my 27 tests."
>356 Petroglyph: "... I'm not sure I follow. Could someone who's more mathy than me explain?"

Happy to help. 1 times 26 = 27. Basic math. I failed an introductory math course at University twice, and even I know that.

>351 lilithcat:

It is as if faktorovich hasn't heard of weekly rep - you basically get a week between the first read-through and first performance.

>359 faktorovich: "Proof by Authority"?

Mathematicians have classified quite a few of those kinds of proof:
https://www.cantorsparadise.com/methods-of-proof-for-every-occasion-1d114c6fe628

BTW the Shakespeare play discussion brought into memory a very funny book I have:

Ben Ross Schneider: Travels in computerland : or, Incompatabilities and interfaces : a full and true account of the implementation of the _London stage_ information bank (1974)

It describes the process of turning the 8000 printed pages of "The London Stage" into a database in 1974. As the author says at one point: "I do not lead a life of quiet desperation; I lead a life of loud desperation."
Many of the problems of turning manuscripts into data that's been mentioned in our discussion here are also touched upon in Schneiders book.

>309 Petroglyph: made me wonder if I could detect the interpreter influence in a large corpus of translated texts by the same author. I was thinking about the over hundred Agatha Christie novels / short story collections translated into Danish by a smallish number of translators. Luckily I don't have the texts available in machine-readable format, so i'm using my lunch break to just eat lunch :-)

>355 faktorovich:

But as a woman without any connections; I am not going to get any funding to develop this program or to market it to researchers. And even if it was the most brilliant program in the market; researchers like you would consider my gender a disqualifier and would refuse to use it even if it was free.

Well, that lets you nicely off the hook, doesn't it?

Look, no one with any sense would deny that there is a gender discrimination in most professions, particularly in STEM fields. But to assume, without evidence, that any and all decisions to reject your work will, in the future be because of your gender hurts not only your credibility, but it hurts all women. Not every denial of funding to a woman or rejection of her work is due to her gender. Sometimes it's because the work doesn't deserve it. You're being very much like the boy who cried wolf.

(By the way, what makes you assume that Petroglyph is a man?)

>356 Petroglyph: "Your PhD in English and LitCrit makes you an expert in zero other disciplines."

Don't be so generous. She's demonstrated that her knowledge of Literature & its criticism is also zero. Criticism doesn't just demand that a thesis be defendable, it demands that it be reasonably defendable. There is no support, literary, historical, or bibliographical, for the statements she's making. As I said in one of my first posts, if one of my first-year students came to me with work like this I'd be shocked at their utter inability to understand how logic works. The fact that this author is gleefully, willfully ignorant about the texts she "analyzes" doesn't make this any kind of lit crit at all.

I'm not sure what academic genre it would fall into. Maybe psych, if someone was looking at the ways her work & posts in this thread are absolute confirmations of the Dunning-Kruger effect.

>365 susanbooks:
Well, what I overuse in generosity there I lack elsewhere, I guess.

I want to be careful when discussing credentials -- it's often a sore point and focusing on it would just lead the discussion down unhelpful pathways.

It's definitely Criticism (there's some weird corners of that field!). One former colleague of mine publishes regularly in Anthropoetics. Some of that is straight LitCrit. Some of it is... Well, I'll hold my tongue.

AnnieMod, I hope you don't mind that I'm using one of your questions as a jumping-off point for another Lunch Break Experiment (tm), although this time it's more "Petroglyph reads a paper and skips the bits that go beyond them."

>354 AnnieMod: "punctuation comparison used to note similarities in style just does not sound like something that would work"

There's a lot of counterintuitive things that actually do give you promising and workable results. For instance, dividing a text into n-grams of n characters or n words. At the bottom of this page, they split a sentence into "3-grams": bunches of 3 characters. And when you use this to try and separate various authors, you get surprisingly accurate results! (I think I could retest the Wizard of Oz books like that and get pretty good results.) It's not a perfect method, by all means, but it delivers more than you'd expect. Worth looking further into, in other words.

But about punctuation. It matters of course exactly what you look at -- mere frequency, proportion of commas per period, sequences of punctuation marks, average number of words between two punctuation marks. I dimly remembered some papers and handbooks talking about it, so I checked my uni's library. And here, published this September, I found this paper:

Darmon, Alexandra N. M., Marya Bazzi, Sam D. Howison, and Mason A. Porter. 2021. ‘Pull out All the Stops: Textual Analysis via Punctuation Sequences’. European Journal of Applied Mathematics 32 (6): 1069–1105. https://doi.org/10.1017/S0956792520000157.

(If anyone wants a pdf, let me know).

So I tried to read it during lunch, and, while much of it went over my head (it was published in an Applied Mathematics journal; I'm a mere linguist), I can share some of the authors' comments and some screenshots of their results, since they talk about topics that have come up in this thread: punctuation as a means of assessing author style, the role of editors in punctuation, the role of online text editions. (I apologize ahead of time if I over-explain things that are obvious to more math-oriented ppl here, or if I make egregious errors. Do tell me if I'm wrong!)

From the abstract:

In our investigation, we examine a large corpus of documents from Project Gutenberg (a digital library with many possible editorial influences). We extract punctuation sequences from each document in our corpus and record the number of words that separate punctuation marks. Using such information about punctuation-usage patterns, we attempt both author and genre recognition, and we also examine the evolution of punctuation usage over time. Our efforts at author recognition are particularly successful. Among the features that we consider, the one that seems to carry the most explanatory power is an empirical approximation of the joint probability of the successive occurrence of two punctuation marks.

Emphasis mine. The bolded part means that they got better results in authorship attribution and genre allocation etc. when they looked at sequences of punctuation marks rather than individual ones.

They used "a corpus of 651 authors and 14,947 documents from the Project Gutenberg database (p. 1072), obtained via their API. Authors' birth dates range from the early 1500s through the late 2000s around 2000.

"We do not attempt to distinguish between an editor’s style and an author’s style for the documents in our corpus; doing so for a large corpus in an automated way is a daunting challenge, and we leave it for future efforts." (p. 1070)

So they do note that editors can have an influence on this particular metric, and that, in a perfect world, could be lifted out and considered separately. This is something they might work on next. They are aware that the problem exists, though, and take it into consideration. You'd need plenty of texts by different authors that have all been touched by the same editor. And you need that sort of corpus for many, many editors. Just the compilation of such a corpus is a labour-intensive operation.

The current study deals with this:

However, punctuation varies significantly across individuals, and there is no consensus on how it should be used (...); authors, editors and typesetters can sometimes get into emphatic disagreements about it.2 Accordingly, as a representational system, punctuation is not standardised, and it may never achieve standardisation. (...) It is plausible that an author’s use of punctuation is — consciously or unconsciously — at least partly indicative of an idiosyncratic style, and we seek to explore the extent to which this is the case.(p. 1072)

Now, this is a serious academic study, so Darmon et al. spend lots of time telling you exactly what they are trying to do, how they did it, and how they solved their methodological problems. So the next section may drag a bit.

Also, this is just an initial methodological study: Darmon et al. are trying to develop a workable way of checking punctuation measures in an automated fashion, if the corpus they've collected is appropriate, and, well, see if it works. This is a good basis on which to develop future iterations of this model.

I've bolded some bits in the next quote. This is a proof-of-concept type of study. Darmon et al. want to develop certain mathematical implementations of various ways to look at the patterns of punctuation. It's not the be-all and end-all of Stylometry.

We do not seek either to try to identify the best set of features or to conduct a thorough comparison of different machine-learning methods for a given stylometric task. Instead, our goal is to give punctuation, an unsung hero of style, some overdue credit through an initial quantitative study of punctuation-focused stylometry. To do this, we examine a small number of punctuation-related stylometric features and use this set of features to investigate questions in author recognition, genre recognition and stylochronometry. To reiterate an important point, we do not account for the effects of editorial changes on an author’s style, and it is important to interpret all of our findings with this caveat in mind (p. 1073)

These next two quotes are from the Data and Methodology section, where the authors tell you how they collected their data, and what choices they made in keeping/rejecting some texts. They also tell you what software they used.

We retain only documents that are written in English. (A document’s language is specified in metadata.) We remove the author labels ‘Various’, ‘Anonymous’ and ‘Unknown’. To try and mitigate, in an automated way, the issue of a document appearing more than once in our corpus (e.g., ‘Tales and Novels of J. de La Fontaine – Complete’, ‘The Third Part of King Henry the Sixth’, ‘Henry VI, Part 3’, ‘The Complete Works of William Shakespeare’ and ‘The History of Don Quixote, Volume 1, Complete’), we ensure that any given title appears only once, and we remove all documents with the word ‘complete’ in the title. Note that the word ‘anthology’ does not appear in any titles in our final corpus. We also adjust some instances in which a punctuation mark or a space appears incorrectly in Project Gutenberg’s raw data (specifically, instances in which a double quotation appears as unicode or the spacing between words and punctuation marks is missing), and we remove any documents in which double quotations do not appear. Among the remaining documents, we retain only authors who have written at least 10 documents in our corpus. For each of these documents, we remove headers using the Python function STRIP_HEADERS, which is available in Project Gutenberg’s Python package. This yields a data set with 651 authors and 14,947 documents (p. 1074)

Ooh, neat: there's a Python package specifically for handling Gutenberg texts. If I ever pick up Python, this is something I want to play with.

Here are the eleven punctuation marks Darmon et al. looked at: . , : ; ( ) ? ! " ... '
Again, they are careful to tell you what they included and what they excluded, and why. Openness and transparency!

For each document, we extract a sequence of the following 10 punctuation marks: the period ‘ . ’; the comma ‘ , ’; the colon ‘ : ’; the semicolon ‘ ; ’; the left parenthesis ‘ ( ’; the right parenthesis ‘ ) ’; the question mark ‘ ? ’; the exclamation mark ‘ ! ’; double quotation marks, ‘ “ ’ and ‘ ” ’ (which are not differentiated consistently in Project Gutenberg’s raw data); single quotation marks, ‘ ‘ ’ and ‘ ’ ’ (which are also not differentiated consistently in Project Gutenberg’s raw data), which we amalgamate with double quotation marks; and the ellipsis ‘ ... ’. To promote a language-independent approach to punctuation (e.g., apostrophes in French can arise as required parts of words), we do not include apostrophes in our analysis. We also do not include hyphens, en dashes or em dashes, as these are not differentiated consistently in Project Gutenberg’s raw data, and we find the choices between these marks in different documents — standard rules of language be damned — to be unreliable upon a visual inspection of some documents in our corpus. Lastly, we exclude square brackets (which are also sometimes called ‘brackets’), as they are used in metadata within the documents in Project Gutenberg. (p. 1075)

Here are the exact usage patterns they measured:



In other words:

• f1: how often does each mark occur?
• f2: If you get a punctuation mark X, how likely is it that Y will be the next mark (with or without intervening words)? (fill in any of the eleven for X, and any of the others for Y; Darmon et al. test all possible combinations)
• f3: how likely is any XY pair (with or without intervening words)? (fill in any of the eleven for X, and any of the others for Y)
• f4: average words per sentence. (where a sentence is taken to end with . ! ? ...)
• f5: average number of words between two punctuation marks
• f6: for a pair XY, how many words, on average, occur between them? (fill in any of the eleven for X, and any of the others for Y)

Faktorovich does F1 for these five punctuation marks , : ; ? !
She does F4, too. The others are not part of her method (because the online tools aren't equipped for them. In fact, I wonder how they define "end of sentence").

Ok. I'll skip over most of the actual methodology, in which they mathematically operationalize these 6 ways of looking into because I must admit that the mathematics are way beyond me:



And this is just the beginning. They also employ Kullback-Leibler divergence, Shannon entropy, and others. I don't know what any of these are. There's a great deal of talk about how they use neural networks, too. Interesting, I'm sure, but I'm lost here.

Again, if you want the pdf to check for yourself, let me know.

Anyway: here are their results for how their measures f1 through f6 correctly separate authors (open in separate tab for larger version):



The percentages in each of the f-columns are how many texts this test correctly assigned to the correct author. Some measures start at 80-90% correct separation of 10 authors; one starts at around two thirds. As more and more authors and texts are included the results become less reliable. Darmon et al. also show what results you get if you test f1, f3, f4 and f5 together, and if you take all 6 together. Better results here than for each of the features taken separately.

To be clear: f1, f2, f3, f5 and f6 each run on all eleven punctuation marks simultaneously. To translate this in Faktorovich units: her 5 punctuation marks and sentence length (which is f4 here) count as 6 of her 27 tests; so f1 counts as 11 tests, and then f2, f3, f5 and f6 count as whatever the number is for any possible pairing between 11 items (each!).

Darmon et al. also look at whether their f1 through f6 can be used to tell various genres apart. Here are the results for that set of experiments (open in separate tab for larger version):



Not great. Not terrible, either, but not all that great. The highest score they have is 65%, for measure f3.

Finally, Darmon et al. also look at changes in punctuation over time (I've hyperlinked their Figures 13 and 14):

We show the evolution of punctuation marks over time for these 616 authors in Figure 13 and Figure 14, and we examine the punctuation usage of specific authors over time in Figure 15. Based on our experiments, it appears from Figure 13 that the use of quotation marks and periods has increased over time (at least in our corpus), but that the use of commas has decreased over time. Less noticeably, the use of semicolons has also decreased over time. In Figure 14, we observe that the punctuation rate (which is given by formula (2.6)) tends to decrease over time in our corpus. However, this observation requires further statistical testing, especially given the large variance in Figure 14. Because of our relatively small number of documents per author and the uneven distribution of documents in time, our experiments in Figure 15 give only preliminary insights into the temporal evolution of punctuation, which merits a thorough analysis with a much larger (and more appropriately sampled) corpus. Nevertheless, our case study illustrates the potential for studying the temporal evolution of punctuation styles of authors, genres and literature (and other text). (pp. 1092-3)

Darmon et al. note that these trends over time have to be taken into account for future studies. If the use of commas and semicolons indeed trends downwards over time, and quotation marks go up, then you can't use the exact same calibration for texts from the 1500s as texts from the 1900s or the 2000s.

Ok, so what do Darmon et al. have to say about their results in general? In their own words:

One feature, which we denoted by f3 , measures the frequency of successive punctuation marks (and thereby partly accounts for the order in which punctuation marks occur). Among the features that we studied, it revealed the most information about punctuation style across all of our experiments. It is worth noting that, unlike the feature f 2 , which also accounts for the order of punctuation marks, f 3 gives less weight to rare events and more weight to frequent events (see equation (2.3)). This characteristic of f 3 , in concert with the fact that it accounts partly for the order of punctuation marks, may explain some of its success in our experiments.

They had most success with f3 ("how likely is any pair of punctuation marks?"): looking at the order in which an author uses punctuation marks seems to be more revealing of their style than taking measures based on how frequently she uses them. (though the latter also gives ok-ish results. But sequences appear to do better.)

They also add this:

It would be interesting (although daunting and computationally challenging for someone to do it with Project Gutenberg) to try to gauge whether and how much different editors affect authorial style. (p. 1095)

and this:

"It is also worth reiterating that Project Gutenberg has limitations with the cleanliness of its data. (See our discussion in Section 2.1 for examples of such issues.) These issues may be inherited from the e-books themselves, and they can also arise from how the documents were entered into Project Gutenberg. Although we extensively cleaned the data from Project Gutenberg to ameliorate some of its limitations, important future work is comparing documents that one extracts from Project Gutenberg with the same documents from other data sources." (p. 1095)

In their conclusions, Darmon et al. mention that further developments of these methods could be applied to different translations of the same work, or of different editions of the same text, as well as trying to tease out "the effects of an editor’s or journal’s style on documents by a given author (an especially relevant study, in light of the potential to confound such contributions in corpuses like Project Gutenberg)" (p. 1096)

Ok. Here ends today's Lunchtime Reading Hour with Petroglyph (tm). And here, at the endpoint of my ramblings, I can't resist pointing out this paper:

Whissell, C. (1996) Traditional and emotional stylometric analysis of the songs of Beatles Paul McCartney and John Lennon. Comput. Human. 30, 257–265.

This looks like fun!

>364 lilithcat: (By the way, what makes you assume that Petroglyph is a man?)

My writing style is probably more similar to that of male LT commenters than to that of female LT commenters.

>368 Petroglyph: And, as Barbie says, Math is hard! -- therefore, by the sort of proof displayed here, I say you're a ghostwritten man.

>367 Petroglyph: The one time in my life when a sentence I write is taken kinda out of context and I don’t mind (because then the explanation proves the context). I love math on Saturday morning. :) What a surprise - editors are important when you study distribution of the punctuation. :)

PS: I’d love that PDF - not because I want to check your work but because it made me curious.

>369 susanbooks:, >370 AnnieMod: Yeah, I'm interested too. Since it was published in Applied Math, I was pretty sure I could find a preprint on arXiv.org, a massive preprint database for physics/math/comp sci/etc. Sure enough, here it is!

https://arxiv.org/abs/1901.00519

And thanks to Petroglyph for starting me down a fascinating rabbit hole - long time R user, but never for this sort of thing...

>370 AnnieMod:
Here's the pdf!

>367 Petroglyph: Markov chains are great fun (and part of a rather large statistics course that I was required to take). The basic idea is that you have a number of states (or nodes in a graph or whatever) and probabilities for going from one state to another.
So the matrix they generate are just these probabilities.

I've used Markov chains to generate random programs for testing a compiler (not Python though).

People have also used it to generate random text with the same patterns as say French.

Thanks for sharing the study.

>369 susanbooks:

Who's to tell who my ghostwriter is? I could be a woman born in the 1930s, who went on to be one of the original computers in the 1940s and 50s (back when tedious data input was done by women). I wonder how many kilogirls this thread is worth by now ;)

>373 bnielsen:
Thanks for the explanation. I can see the, well, what I call the general shape and structure of how that works, but I'd have to spend some actual thinking time with a tool/book/person to feel semi-confident in talking about them.

>374 Petroglyph: I'm guessing it must be even older than that. A petroglyph must surely be from the stone age, so maybe all text is written by a single ghostwriter? (I think we are heading towards solipsism-by-proxy.)

Darmon-et-all style: .,?(.)

Life's too short...

>376 bnielsen:
I wonder who we could name that condition after...

>367 Petroglyph: The frequency of different punctuation marks is a highly precise method of authorial-attribution. This is why I used it as a significant component in my method, so this article’s main experiment re-enforces the strength of my method. The tests applied in this article are not relevant for the “Lunch” experiment Petroglyph has included because he tested for word-frequency and not for punctuation frequency. The most frequent letters and words are also very accurate (based on the hundreds of texts I have tested), but splitting all letters into non-computable 3-letter and 3-word etc. clusters is not at all relevant or rational for authorship-assignment.

"mere frequency, proportion of commas per period, sequences of punctuation marks, average number of words between two punctuation marks." Imagine if you were attempting to solve a murder. The "mere frequency" test is equivalent to merely testing the DNA left at the scene and its frequencies or markers against suspects; the alternative "portion of commas per period" would be like testing the DNA against the finger-prints - it is a good idea to do both, but why compare them to each other if both provides accurate results; and this method is not only asking to compare them put to establish the quantity of DNA left vs. the size of the finger-prints left; making this comparison of oranges to apples is nonsensical because it digresses away from the murder mystery at-hand into playing with math equations to give more work-hours to the technician/ researcher on the case. For example, "average number of words between two punctuation marks" is already calculated in my tests by the separate tests for the average number of words per sentence, and the frequency of the various significant punctuation marks; these elements are already combined when I incorporate these 27 different tests into the final results, so also measuring the number of words between the marks adds a test to the analysis that repeats the findings of several of the already accounted-for tests. The term the article uses for this test-doubling is " joint probability" - this is a statistics/investment term that means checking the probability of 2 events happing together etc. This measure is significant in finance to estimate the behavior of stocks based on 2 different events happening that might influence their direction. In linguistics, the texts being tested have already been written and there is nothing unknown about what words/ punctuation they might include. Therefore, the testing of elements like punctuation and words-per-sentence or words-per-paragraph are more accurate when they are performed separately; and recording the data of these tests separately helps researchers to figure out if specific punctuation marks or specific sentence/paragraph length contributed most significantly to a given attribution; this data helps to figure out the impact each element has in relation to genre/ period etc.; when these tests are combined into a formula and the separate measurements or data-points are not provided, the researcher does a disservice to readers interested in further exploring the results.

"We remove the author labels ‘Various’, ‘Anonymous’ and ‘Unknown’." This step can be used to remove half of the Renaissance texts because over half of them were initially anonymous. So this elimination can be a serious bias that can skew the results of this study, as it would not recognize similarities between "Shakespeare" and otherwise bylined texts from these decades and these anonymous etc. texts, some of which were significantly first-published after "Shakespeare's" death.

Another problem, "we remove any documents in which double quotations do not appear": this can be used to exclude all old-spelling versions of texts in favor of only heavily edited modern editions that have been standardized with double quotation marks. As I explain this becomes essential for attribution in the Renaissance where the old-spelling/old-punctuation version of the anonymous "King Leir" does not match the modern-spelling version of the "Shakespeare"-bylined "King Lear", but does match the old-spelling "Shakespeare"-bylined version of "King Lear". The heavy translation process that has been applied to modernize Renaissance texts makes them incomparable to originals, and if different editors were involved not-comparative to each other in these modern versions either.

Yet another problem, "we retain only authors who have written at least 10 documents in our corpus"; as we have discussed in this chat before, the texts by authorial-bylines with under 10 documents to their names are the most likely to have been the work of ghostwriters, so establishing if these match texts with other bylines should be the primary concern of researchers attempting to test an enormous corpus for attribution. The exclusion of these under-10 text writers again skews the results towards re-affirming the bylines of the prolific writers without recognizing similarities between their styles and possibly some of these excluded minor "writers".

I already pointed out the basic unnecessity of testing punctuation in combination with word count and in random combinations with other punctuation marks. My method achieves the same combination of these different measures that is beneficial for attribution, without each sub-measure being duplicated in several formulas that measure convoluted combinations before being averaged out in the final attribution. Only somebody that deliberately wants to prove attribution with computational-linguistics is impossible would calculate the odds of every occurrence of punctuation mark X after a Y punctuation mark. The simple frequency of X and Y established the authorial-style, whereas these combinations can generate wildly random results, and especially so if all of these different combination possibilities are combined into a single attribution formula. I do not use F 2-3 and F 5-6 in my method not because I lack the tools to apply them, but because there is no added level of attribution certainty to performing these mathematical magic tricks that are only for show, and not for substance. For example, the “number of words between two punctuation marks” is the same as measuring the length of sentences and the frequency of commas; the tests of comma and period frequency give the same basic measure of words between these punctuation marks.

You have cut off the portion of the article where elements of the main equation for punctuation-marks-matrix is defined. If you have read math-papers before, surely you would recognize that including the final product of a formula, and not its components, makes it impossible to criticize the errors in the steps. I do not have access to the full-paper, so I cannot fully explain what is wrong with this final formula. And not knowing terms like “Kullback-Leibler divergence” is easily solved by looking them up = it is basically a comparative analysis of statistical distribution. “Shannon entropy” refers to uncertainty vs. certainty of the output. Neither of these concepts are needed to attribute authorship, as my simple method proves. Uncertainty and the full range of distribution become problems when the mathematical equations applied are unnecessarily convoluted so that they themselves introduce uncertainty or chaos instead of just solving the attribution mysteries they should be designed to solve. If you shared the pdf with me, and I explained the precise mathematic errors in their approach; you would clearly not pay attention to my explanation, as you are skimming through their explanation, and they are researchers who you respect.

Their “Accuracy for the testing set” measures the percentage of cases where the byline attributions in the texts matched their authorial-signature assignment. It indicates that they achieved the highest degree of accuracy on test f3 when only testing 10 different bylines, f1 was second-best, and f5 third, and f4 fourth. Yet they are suggesting combined these texts together into an average leads to an overall-accuracy that is close to the top of the four f3 (93%) vs overall 89%, when f4 was only 64%; a simple average of the 4 measures equals 82%, so they have tinkered even with this average measure. These findings also mean that one of their more convoluted tests for punctuation pairs was the most accurate test. While one of the tests I have found to be especially reliable (words-per-sentence) is the least accurate test. Of course the problem with the words-per-sentence test alone is that different authors can have similar words-per-sentence frequencies, but they can easily be distinguished on other measures included in my 27-tests method; this is why I do not use any one of these tests. This article’s method also proved to disintegrate when the corpus was expanded to 100 authors (my corpus for the Renaissance had 104 authors) – in this corpus, 3 out of 4 of the measures were at 50/50 chance of accuracy or lower, with one as low as 37%; but the combined f1, f3, f4, and f5 average remained at an irrationally-high level of 79%. The bigger problem is that accuracy of attribution might have been increasing as the number of authors grew, while only the byline accuracy was decreasing; in other words, the giant corpus might have finally allowed the researchers to accurately determine which of the texts shared linguistic signatures or had single underlying ghostwriters, but these matches are dismissed as inaccuracy because all divergences from the stated bylines are judged to be errors.

On the genre-experiment, their result was that computational-linguistics tests they applied cannot be used to distinguish between genres. This is exactly correct. Any accurate attribution method should not impacted by the genre used in the texts. Genre should be considered by literature scholars to understand how it impacted the text’s data, but authorial-attribution works outside of genre-lines. My method found multi-genre texts that matched a single authorial-signature for all of the Renaissance ghostwriters.

The graphs do not make any rational sense. Graph c has frequencies between 0 and 9%, d up to 4%, but a and b stretch between 0 and 60%. Why would the punctuation rates be 10-times higher in a and b vs. c and d for the same centuries? And this graph does not have a definition for the different line colors, so it is unclear if exclamations or semicolons usage is going up or down.

Overall this article is a great example of misleading, deliberately convoluted, and obviously erroneous methodology that is the reason this field needs to be fixed by instead applying my accurate attribution method. The fact that data has been manipulated is apparent from the broad findings. They do not present the raw data and the data for the steps of their calculations or the precise f1-6 data points before these were incorporated into these seemingly simple summary-tables and visuals. The raw data is not likely to be anywhere as neat and orderly, and I would assume that if I examine the exact calculations involved in each step of this process, I would find the specific errors and apparent data manipulation involved that has led to these results that seem to represent a simple relationship between corpus-size and byline re-affirmation accuracy. My own testing contradicts this result as many bylines share linguistic-signatures when a ghostwriter is involved etc. I could create several colorful graphs to explain my method, and could phrase my method in terms of formulas, but this would do nothing but confuse the public who might just need the basic steps to reproduce or test my results.

>379 faktorovich:
"average number of words between two punctuation marks" is already calculated in my tests by the separate tests for the average number of words per sentence, and the frequency of the various significant punctuation marks; these elements are already combined when I incorporate these 27 different tests into the final results, so also measuring the number of words between the marks adds a test to the analysis that repeats the findings of several of the already accounted-for tests"

This comment, Ladies and Gentlemen, makes me giddy.
Faktorovich tests the absolute frequencies of five punctuation marks , : ; ? !. Let's forget that Darmon et al. do 11, and just entertain this for a second. Sure. Darmon et al test at least five, too.
A separate test that her online tools perform is average sentence length. This, Darmon et al. do, too.
Faktorovich imagines that, merely by testing these things separately and independently, and then by obliterating their independent results into a binary oppositon (for ease of counting), that she also has tested their interaction.

Holy fuck, you guys.

No wonder she feels she's gone beyond the laws of probability. (>305 faktorovich: "The more I apply computational-linguistic tests to texts and research these authorship mysteries, the more convinced I am that there are no coincidental matches; all matches indicate shared authorship.")

At this point I feel it's necessary to ask: are we sure that the person posting under this moniker really is Dr. Anna Faktorovich? Are we really sure it's not some troll going around posting mindblowing nonsense under her name to make her look bad?

Holy Dunning-Kruger, Batman!

Doktor Faktorovich, I stand in awe at the depths of your misunderstanding. I must thank you for bringing this unexpected jolt of mirth into my life.

I'm sure I'll have more to say later -- I haven't made it further than that quote, tbh. Truly, I stand in awe, and I wanted to take a minute to live that moment to the fullest.

I wonder what the result would be when comparing the posts in this thread by faktorovich and Petroglyph using faktorovich's method.

>381 igorken:

I think we'd find that they were all actually written by Tim Spalding.

So this what becomes of Matilda and other children that Roald Dahl writes about: they grow up into adults with the doggedness and the sheer mental force to reject reality and substitute their own. I genuinely mean that as a compliment, by the way, no joke, no tongue in cheek: I love Matilda.

Do you read fiction at all, Dr Faktorovich? If so, you might enjoy Angel by Elizabeth Taylor, or even Hadrian the Seventh by Frederick Rolfe. Both these books are about people like yourself: strong-willed characters who believe so strongly in their own version of reality that the World At Large has no choice but to yield. (I've written reviews of both of these.) Or if you're more into biographies, I can also recommend The Quest for Corvo: an experiment in biography by A. J. A. Symons -- a biography of Frederick Rolfe that is... *chef's kiss* ... absolutely riveting. (My review here.)

I've now also sent you a DM asking for a pdf copy of your book The Re-Attribution of the British Renaissance Corpus. I'll be happy to read it over the end-of-year break.

>382 lilithcat:
So *that* is why changes to the site always take "two weeks"!

>383 Petroglyph: I just started putting together the 15 book reviews I am writing for my Pennsylvania Literary Journal. One of these is fiction. I requested all except for the fiction title from these publishers. I tend to only read fiction when I am researching it for an article/book, or reviewing it. In the last few years (before a downturn in the number of review copies publishers send) I have been reviewing around 150 book titles per year for PLJ. And I read thousands of books annually of all types as I work on my research. Given that research has been my full-time job for most of the last 10 years, the line between "fun" reading and professional reading has blurred. I wrote a book about formulaic writing, "Formulas of Popular Fiction", where I explain my views on most pop genres/ pop books. I do not escape into books to find myself, but rather read books to find the underlying truth about their authors. There is nothing fantastical about my Re-Attribution series. It might seem fantastical if you read my summaries of its volumes and the re-attribution conclusions, but as you should find out as you review the series itself, it presents overwhelming concrete evidence for the version of history my re-attributions indicate.

>385 faktorovich:
That's ok. I don't judge people for their reading habits.

Also, I'm not suggesting that your methods are fantastical or fictional. I'm saying that they're nonsense.

>379 faktorovich:
The most frequent letters and words are also very accurate (based on the hundreds of texts I have tested), but splitting all letters into non-computable 3-letter and 3-word etc. clusters is not at all relevant or rational for authorship-assignment."

Ok, I'm sorry, everybody, but I've got to explain how completely misguided this is.

Let's take that example of a character ngram where n=3 from this page:

“Happy families are all alike; every unhappy family is unhappy in its own way.”

Split this into strings of three characters, or trigrams, and you get this (a space is also a character):

"h a p" "a p p" "p p y" "p y " "y f" " f a" "f a m" "a m i" "m i l" "i l i" "l i e" "i e s" "e s " "s a" " a r" "a r e" "r e " "e a" " a l" "a l l" "l l " "l a" " a l" "a l i" "l i k" "i k e" "k e " "e e" " e v" "e v e" "v e r" "e r y" "r y " "y u" " u n" "u n h" "n h a" "h a p" "a p p" "p p y" "p y " "y f" " f a" "f a m" "a m i" "m i l" "i l y" "l y " "y i" " i s" "i s " "s u" " u n" "u n h" "n h a" "h a p" "a p p" "p p y" "p y " "y i" " i n" "i n " "n i" " i t" "i t s" "t s " "s o" " o w" "o w n" "w n " "n w" " w a" "w a y"

It has been demonstrated that, just like relative word frequencies can help in distinguishing two authors' bodies of work, trigrams (n-grams of length 3) can do so, too. You get pretty good results. Not perfect, but pretty good! And I think I that I can also convincingly argue that trigram frequencies can, indeed, correctly attribute an author's work to them.

First, I'll show you. Here is a cluster graph for the Oz novels in >309 Petroglyph:, generated using character n-grams of length 3 (with the pronouns left in):



All of Baum's novels cluster together, all of Thompson's novels cluster together, and the mystery book suspected to be authored by Thompson (which was confirmed via several stylometric analyses) has also been judged to be extremely similar to the Thompson books. Just like in >309 Petroglyph:, where word frequencies were used.

So, empirically, character trigrams and single-word frequencies give similar results. (Anyone arguing that this method is faulty must also find a reason for why they're faulty in exactly the same direction.)

Second, I'll explain the theory behind why this works.

Alright. An "n-gram" is simply a "gram" -- a connected string of characters -- of length n.

We humans tend to think in units of meaning, and for English those include (but are not limited to) words, phrases, sentences, chapters, novels, trilogies, ... But computers don't think like that. To them, a character, a word, a phrase, a sentence, a paragraph, a chapter, a novel, a trilogy, a corpus ... are just strings of characters of different lengths. They're the same category of thing, just of different lengths. And you can totally use strings of different lengths to distinguish two authors from each other. Even Faktorovich agrees with that. Here are some examples:

• If your method looks at the frequency of three-word phrases, it looks at the frequencies of word n-grams of length 3. These can be useful in authorship attribution
• If your method looks at single word frequency, it looks at word n-grams of length 1. These, too, can be useful in authorship attribution.
• If your method looks at single character frequencies, it looks at character n-grams of length 1. These, too can be useful in assigning authorship
• A character n-gram of length three is not some totally different category from single characters and words and phrases. It is merely the logical observation that, given the previous bullet points, strings of characters that are longer than 1 but shorter than the average word can also be useful in authorship attribution

A machine doesn't care if it is told to chop the long long string of characters that is a novel into various pieces of uneven length at the spaces and punctuation marks (~ words), or into even pieces of three characters (trigrams), or into even pieces of single characters (character n-grams of length 1, aka single letters & punctuation). It just does what it's told.

The reason that relative frequencies of character trigrams can be useful in assigning authorship is exactly the same reason why individual word frequencies can. This would be obvious to someone who's been doing stylometry research for years. Right? Right?

Dr Faktorovich. If you really, truly, genuinely believe that looking at n-grams is "non-computable" and "not at all relevant or rational for authorship-assignment", then I recommend you read The Re-Attribution of the British Renaissance Corpus, by Dr. Anna Faktorovich. That book contains Volumes 1-2 of her Re-Attribution Series, and in its densely argued 698 pages, she argues for this infallible and extremely sophisticated computational-linguistic method that she's invented. This method makes extensive use of word n-grams of length 3, word n-grams of length 1, and character n-grams of length 1.

>387 Petroglyph: If you split all of the letters into a text into trigrams, according to your example, you would be multiplying the quantity of letters in the test-sample by at least 6 and each of these X6 portions would be measuring basically the same letters but in different combinations. Retesting the same letters six times in different combinations is the definition of nonsensical research. My method demonstrates that just testing the most frequent letters and words is the rational approach. Your diagram can be a creative art project without any data to support it. A diagram that shows "correct" or byline-affirming attributions does not prove it is correct; the errors I have been explaining are not in your or other researchers' in this field capacity to draw diagrams that make them seem believable, but rather in your erroneous method and the lack of disclosed underlying raw data.

Your n-grams for 3-letters example is not for "different length". If they were "different length" this would just break the rules of the experiment of test 3-letter n-grams, but it would do nothing to establish attribution. Measuring the most frequent units (words, letters, phrases) is most useful for attribution. The 3-word phrases are indeed 1 example of n-grams or strings that I use, but I do not compare these phrases quantitatively, but rather use them to verify if the quantitative tests match any pattern that emerges between these phrases. I am not arguing against the testing of all strings of more than one unit, just again any method that repeats the testing of the same units with the same tests by creating multiple nonsensical combinations of these units. The term n-grams can signify a rainbow of different things, so it is best to avoid it and instead to state if what is being compared is words, letters, phrases etc. to be specific in a research article; instead scholars in this field use the term n-gram especially when they do not want to reveal that they are only testing for word-frequency, as saying n-gram instead can lead a reader to assume something more "advanced" is actually being tested.

If as you say "relative frequencies of character trigrams" are "exactly" as useful as "individual word frequencies"; then, you have lost any reason to object to my method. You have been saying that the only real error in my method is that it does not use these "relative frequencies of trigrams"; so if my method is just as statistically applicable, it is equally as correct according to this principle. The problem is that the linguistic paper we are now discussing (and others like it) use the complexities of dividing texts into many strange parts, and various other tricks to confuse casual readers so they do not notice that the data to support the conclusions or the pretty final diagrams is missing, or the provided data are summaries statistics and do not show where this data came from or how it was processed to get the final summary-results.

>383 Petroglyph: I'm in the middle of Angel, and yes, the resemblance is striking.

>388 faktorovich:
"you would be multiplying the quantity of letters in the test-sample by at least 6 and each of these X6 portions would be measuring basically the same letters but in different combinations. "



... Multiplying? I don't split letters into trigrams. I just tell the software to look at strings of three characters. The characters that are there in the text. No new characters are added, or multiplied, or counted more than once. Where do you get these absolutely unfounded ideas? Have you ever thought of starting a cult?

"testing the most frequent letters and words is the rational approach"
Relative frequencies. Normalized against averages. But yeah.

"Your diagram can be a creative art project without any data to support it"
It can also be just a reflection of an honest data analysis. You just don't trust anything that's too far outside your comfort zone. We've been here before.

"Your n-grams for 3-letters example is not for "different length". "
Correct. I said "a character, a word, a phrase, a sentence, a paragraph, a chapter, a novel, a trilogy, a corpus ... are just strings of characters of different lengths. They're the same category of thing, just of different lengths". A word is a string of characters; so is a phrase; so is a sentence; so is a paragraph; and so on. They're all character strings, but a word does not have the same length as a paragraph, a novel, a trilogy...

Really, you're misreading me at such a basic level that my troll-senses are tingling.

"The 3-word phrases are indeed 1 example of n-grams or strings that I use, but I do not compare these phrases quantitatively, but rather use them to verify if the quantitative tests match any pattern that emerges between these phrases."

Lady, you verified Jack Shit.

"relative frequencies of character trigrams" are "exactly" as useful as "individual word frequencies"
Relative word frequencies, yes. I have not changed my position at all.

Look. You keep misreading sentences and misinterpreting them in the most literalist of ways. Is this you trolling us, or is this you being neurodivergent?

"the complexities of dividing texts into many strange parts"
... punctuation is strange parts now?

"various other tricks to confuse casual readers so they do not notice that the data to support the conclusions or the pretty final diagrams is missing, or the provided data are summaries statistics and do not show where this data came from or how it was processed to get the final summary-results"

Your problem, throughout, seems to be that scholarly literature written for experts is hard (or even impossible) to comprehend by non-specialists. That is not a bug, lady, that is a feature. There is value in being able to have conversations at various levels of complexity. Look at Judith Butler's writing. Non-LitCritters bounce off stuff like that. Car mechanics can say things about engines that non-mechanical people are unable to follow. Every profession has this. Maths and linguistics and theology and archaeology and evo-devo and even tree surgeons.

If mathematicians writing to each other had to dumb down their stuff so that non-mathy people could clearly follow everything, there'd be no scholarly literature for mathematics. The same is true for any other advanced field of knowledge -- including car mechanics, electricity, dam construction, cement mixing, glass blowing, sex workers, media criticism and even fanfic writers. With respect to that last one: Are you aware that Drapple is a cargo ship?

This is not done to intentionally shut out non-specialists. In fact, for stats and maths and LitCrit and so many others there is a whole university system dedicated to training interested laypeople into becoming specialists so that they can have advanced conversations with fellow-specialists. If the goal were to shut people out, there'd be no BA, MA or PhD programmes! There's masses of educational materials out there, all free of charge, and of exquisite pedagogial levels.

You claim to have been active in that universisty system at multiple institutions and in more than one country. Are you part of the Great Conspiracy to keep the masses away from arcane knowledge and confuse them?

Look at it from the other direction. This discussion we've been having would be incomprehensible to a seven-year-old. The books you've authored would be. They wouldn't be able to read or understand it, and they haven't got the attention span to keep up. Not every discussion among adults should be held at the level of a 7-year old. Similarly, not all discussions about monte carlo markov chains should take into account the people who don't know what that is.

If you had your way, discussions about any advanced subject would be impossible because of all the sealioning.

Learn what that word means, Faktorovich. I'm going to use it again in referring to you.

You have this fundamental problem with not understanding basic maths, with scholarly literature that you do not understand, and it bothers you so much that you jump to conspirational conclusions.

Neurotypical people don't do this. Neurotypical people understand that different levels of complexity are appropriate (and desirable) in various contexts.

Sealioning trolls pretend not to understand. Several groups of neurodivergent people have trouble understanding this through no fault of their own.

I repeat: either you're a sealioning troll, or you're neurodivergent. Which is it?

By the way, faktorovich. You still owe me an apology for all those lies your lying self told about me.

>389 amanda4242:
Are you enjoying the book?