{"id":5099,"date":"2021-12-18T00:01:52","date_gmt":"2021-12-18T08:01:52","guid":{"rendered":"https:\/\/c-for-dummies.com\/blog\/?p=5099"},"modified":"2021-12-25T07:50:59","modified_gmt":"2021-12-25T15:50:59","slug":"a-tally-of-unique-words-part-ii","status":"publish","type":"post","link":"https:\/\/c-for-dummies.com\/blog\/?p=5099","title":{"rendered":"A Tally of Unique Words, Part II"},"content":{"rendered":"

Continuing with my Unique Words project from last week’s Lesson<\/a>: Once the buffer contains text, the next step is to parse the words: to split the long string of text stored in memory into separate word chunks. For this task, I turn to my old pall, the strtok()<\/em> function.
\n
\nThe process starts with the text file sonnet18.txt<\/a> opened and stored in a buffer. No matter how many lines of text are in the file, it’s stored as one long string, which is perfect for the strtok()<\/em> function<\/a> to slice through. But first some additions must be made to the code.<\/p>\n

Three new variables are declared:<\/p>\n

const char separators<\/code> lists those characters used by the strtok()<\/em> function to parse words from the buffer.<\/p>\n

char *word<\/code> references words found in the buffer, retaining the address\/offset.<\/p>\n

int count<\/code> counts the words found.<\/p>\n

Here is how the updated variable declaration statements for the main()<\/em> function appear when added to the source code file presented last week:<\/p>\n

\r\nconst char filename[] = \"sonnet18.txt\";\r\nchar *buffer;\r\nconst char separators[] = \",.:;!?\\n \";\r\nFILE *fp;\r\nchar *word;\r\nint offset,ch,count;<\/pre>\n
The printf()<\/em> statement at the end of the main()<\/em> function is removed. A while<\/em> loop is added in its place to parse the buffer, count and output the words. These are the new statements added to the code:<\/p>\n
\r\ncount = 0;\r\nword = strtok(buffer,separators);\r\nwhile( word )\r\n{\r\n    printf(\"%3d:%s\\n\",count+1,word);\r\n    word = strtok(NULL,separators);\r\n    count++;\r\n}<\/pre>\n
The strtok()<\/em> function must be called twice. The initial call at Line 54 (that’s Line 54 from the full source code file) identifies the buffer and the characters stored in the separators<\/code> string. The value returned is a pointer to the first word in the string, saved in char<\/em> pointer word<\/code>. When the value returned is NULL, strtok()<\/em> has exhausted the search string.<\/p>\n
The while<\/em> loop spins as long as new words are parsed from the buffer. The strtok()<\/em> function’s first argument replaced with NULL at Line 58 to keep scanning the same string. The output generated consists of a long list of words in the buffer:<\/p>\n
  1:Shall
\n  2:I
\n  3:compare
\n  4:thee
\n  5:to
\n  6:a
\n  7:summer\u2019s
\n  8:day
\n  9:Thou
\n 10:art
\n...
\n109:and
\n110:this
\n111:gives
\n112:life
\n113:to
\n114:thee<\/code><\/p>\n
Click here<\/a> to view the full source code in my GitHub repository.<\/p>\n
At this point the memory addresses saved in the word<\/code> pointer are lost, continuously overwritten. But this problem is okay! The code confirms that words in the buffer can be counted and parsed, which is another step toward finding unique words and those words that repeat.<\/p>\n
Oh, and I don’t free pointers buffer<\/code> or word<\/code> because all allocated memory is released when the program quits. If it makes you feel good, you can add these statements to the end of the main()<\/em> function, before the return<\/em> statement:<\/p>\n
free(buffer);
\nfree(word);<\/code><\/p>\n
Freeing buffers is always necessary when they’re allocated for temporary storage in a function.<\/p>\n
In next week’s Lesson<\/a>, I continue the improvement process by retaining the word pointers allocated.<\/p>\n","protected":false},"excerpt":{"rendered":"
The next step in locating unique words is to parse the buffer, slicing into separate word chunks. Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-5099","post","type-post","status-publish","format-standard","hentry","category-main"],"_links":{"self":[{"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5099","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5099"}],"version-history":[{"count":5,"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5099\/revisions"}],"predecessor-version":[{"id":5126,"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5099\/revisions\/5126"}],"wp:attachment":[{"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5099"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5099"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/c-for-dummies.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5099"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}