Similarly, in your case one might argue that in occurrences where it’s reasonably likely that the capitalisation is due to the word’s position (rather than being a basic attribute of the word like in proper nouns), it makes sense to convert to lower case before processing (e.g. That* or Man*) for upper-case occurrences of words whose dictionary form is lower-case, presumably where context indicated that the upper case was attributable to the position of the word (beginning of sentence or paragraph). ¹ I have seen one corpus-based list carry additional entries (with asterisks, e.g. on the likelihood of occurrence of a particular lemma at the start of an utterance. So here’s an emphatic vote to leave the data unchanged, even if this might mean double entries for many words – which again would also carry potentially useful information, eg. However, restoring the original state from a lower-case list is impossible without external sources (and difficult even *with* external help, for instance dictionaries or spellcheckers). After all, anyone can convert a list to lower case (and recompute frequencies if desired) with minimal effort. However, if you want to re-use the actual words, I’m afraid that forcing all words to lower case might do more harm than good, particularly (but not only) in languages with extensive use of capitalisation like German.Īll other corpus-based word lists l’ve come across so far leave the data untouched¹. If you want to use the frequency data alone (whatever they might be good for on their own) forcing lower case might be fine. I guess it probably depends on what you actually want to do with the lists.

So, it turns out the total words number of your corpus is slightely less than that of COCA which means the total number of word in yours must be around 400,000,000 words. Judging by words occurancy in your corpus, its size cant exceed that one of COCA. so, again: this cant be a true figure - Total word count: 690,318,369,316.

words per file would give a file of 10 Mb as minimum. it appears that this figure of 70 billions words is just incorrect, because a simple calculation such as “Total word count/Total files” gives 6 and a half millions words in a single file, which, you must acknowlege, merely cant be true, especially applying to an average subtitle-file that is supposed to be quite small in terms of the words contained in it.Ħ,5 mln. Dave, i meant the total words figure, of course, that makes up your corpus.

Excuse my french zip download