{"id":1634,"date":"2026-03-06T17:42:12","date_gmt":"2026-03-06T17:42:12","guid":{"rendered":"https:\/\/www.woodcentral.com\/-\/peter\/?p=1634"},"modified":"2026-05-24T11:28:09","modified_gmt":"2026-05-24T11:28:09","slug":"how-the-woodcentral-archive-section-works","status":"publish","type":"post","link":"https:\/\/www.woodcentral.com\/-\/peter\/how-the-woodcentral-archive-section-works\/","title":{"rendered":"How the WoodCentral Archive section works"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Overview<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The system is a <strong>static inverted-index search engine<\/strong> for your legacy PHP articles. It reads <code>.php<\/code> files from the <code>articles<\/code> directory, extracts content, builds <strong>three JSON caches<\/strong>, and enables fast keyword or search-term lookups without querying a database.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The three JSON outputs are:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong><code>archive_meta.json<\/code><\/strong> \u2013 Metadata per file (title, snippet, keywords).<\/li>\n\n\n\n<li><strong><code>archive_keywords.json<\/code><\/strong> \u2013 Frequency count of all meta keywords across files.<\/li>\n\n\n\n<li><strong><code>archive_index.json<\/code><\/strong> \u2013 Inverted index mapping <strong>words \u2192 files<\/strong> containing them.<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Workflow<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1. File Discovery<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Uses <code>glob($articlesDir.'\/*.php')<\/code> to locate all PHP article files.<\/li>\n\n\n\n<li>Iterates over each file and reads the contents via <code>file_get_contents()<\/code>.<\/li>\n\n\n\n<li>Files that fail to read are skipped with a warning.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">2. Metadata Extraction<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Title<\/strong>: Extracted using regex: <code>\/&lt;title>(.*?)&lt;\\\/title>\/si<\/code>.<\/li>\n\n\n\n<li><strong>Meta keywords<\/strong>: Extracted using regex:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>\/&lt;meta\\s+name=&#91;\"']keywords&#91;\"']\\s+content=&#91;\"'](.*?)&#91;\"']\\s*\\\/?&gt;\/si\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Keywords are normalized: lowercased, trimmed, and stored in both:\n<ul class=\"wp-block-list\">\n<li><code>$fileKeywords<\/code> for the current file.<\/li>\n\n\n\n<li><code>$keywords<\/code> global array for counting occurrences.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">3. Content Extraction for Search<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Primary content comes from <code>&lt;main id=\"wc-main\"><\/code>:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>\/&lt;main&#91;^&gt;]*id=&#91;\"']wc-main&#91;\"']&#91;^&gt;]*&gt;(.*?)&lt;\\\/main&gt;\/si\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If <code>&lt;main><\/code> is missing, the <strong>entire file content<\/strong> is used as a fallback.<\/li>\n\n\n\n<li>Content is <strong>stripped of HTML tags<\/strong> using <code>strip_tags()<\/code>, whitespace is normalized with <code>preg_replace('\/\\s+\/',' ', ...)<\/code>.<\/li>\n\n\n\n<li>A <strong>display snippet<\/strong> is created using <code>mb_substr($snippetText, 0, 300)<\/code> \u2014 used in search result previews.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">4. Inverted Index Generation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Full text for indexing: <code>$text = strtolower($title . ' ' . $snippetText);<\/code><\/li>\n\n\n\n<li>Clean-up for search indexing:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>$text = preg_replace('\/&#91;^\\p{L}\\p{N}\\s\\-]+\/u',' ', (string)$text);\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Removes punctuation <strong>except internal dashes\/numbers<\/strong> (e.g., <code>1-qt<\/code>).<\/li>\n\n\n\n<li>Converts all letters to lowercase for case-insensitive search.<\/li>\n\n\n\n<li>Split text into words using <code>preg_split('\/\\s+\/u', (string)$text, -1, PREG_SPLIT_NO_EMPTY)<\/code>.<\/li>\n\n\n\n<li>Words are filtered:\n<ul class=\"wp-block-list\">\n<li>Minimum length: 2 characters.<\/li>\n\n\n\n<li>Stop words excluded (common words like <code>the<\/code>, <code>and<\/code>, <code>for<\/code>).<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>For each valid word, add the filename to the inverted index <code>$index[word][]<\/code>.<\/li>\n\n\n\n<li>After all files, duplicates in <code>$index[word]<\/code> are removed using <code>array_unique()<\/code>.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">5. JSON Cache Files<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong><code>archive_meta.json<\/code><\/strong> \u2013 keyed by filename:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"russ04.php\": {\n    \"file\": \"russ04.php\",\n    \"title\": \"A RECIPE FOR CREATING SPALTED WOOD\",\n    \"snippet\": \"Since the question of spalting comes up time and again, I will share a description of ...\",\n    \"keywords\": &#91;\"russ fairfield\"]\n  }\n}\n<\/code><\/pre>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li><strong><code>archive_keywords.json<\/code><\/strong> \u2013 frequency map:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"russ fairfield\": 1,\n  \"woodturning\": 12,\n  ...\n}\n<\/code><\/pre>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li><strong><code>archive_index.json<\/code><\/strong> \u2013 inverted index:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"oak\": &#91;\"russ04.php\", \"other_article.php\"],\n  \"leaves\": &#91;\"russ04.php\", \"article_521.php\"],\n  \"spalted\": &#91;\"russ04.php\", \"spalt_article.php\"],\n  ...\n}\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enables <strong>fast lookup<\/strong> of files containing any given word.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">6. Search Process (Client-Side Reference)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>User input (<code>search<\/code> parameter) is normalized similarly: lowercase, punctuation removed, split into words.<\/li>\n\n\n\n<li>The inverted index is consulted for each word.<\/li>\n\n\n\n<li><strong>Intersection of arrays<\/strong> ensures that multiple-word searches only return files containing all words.<\/li>\n\n\n\n<li>Keyword filters (<code>keywords[]<\/code>) are applied post-search using <code>array_intersect()<\/code> with the article\u2019s meta keywords.<\/li>\n\n\n\n<li>Final results are sorted alphabetically by title.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">7. Technical Notes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>PHP 8.3 Compatibility<\/strong>: <code>(string)$text<\/code> ensures <code>preg_split()<\/code> never receives <code>null<\/code>.<\/li>\n\n\n\n<li><strong>Unicode Support<\/strong>: Regex uses <code>\\p{L}<\/code> and <code>\\p{N}<\/code> with the <code>u<\/code> modifier for proper Unicode word handling.<\/li>\n\n\n\n<li><strong>Performance<\/strong>: All JSON caches are prebuilt; runtime search is array-based \u2192 avoids database queries.<\/li>\n\n\n\n<li><strong>Extensibility<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Additional stop words can be added to <code>$stop<\/code>.<\/li>\n\n\n\n<li>Snippet length adjustable via <code>mb_substr()<\/code>.<\/li>\n\n\n\n<li>Indexing rules can be modified (e.g., include numbers, punctuation, or phrases).<\/li>\n\n\n\n<li>Files without <code>&lt;main><\/code> are safely indexed using fallback content.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">8. Recommendations for Modifications<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Exact Phrase Matching<\/strong>: Currently the search is word-based. Post-filtering <code>$results<\/code> with <code>stripos($content, $searchPhrase)<\/code> will allow phrase-only matches.<\/li>\n\n\n\n<li><strong>Multi-Language Support<\/strong>: Extend stop words and character classes for non-English letters.<\/li>\n\n\n\n<li><strong>Incremental Indexing<\/strong>: Currently, the script rebuilds everything. Could optimize to only re-index changed files.<\/li>\n\n\n\n<li><strong>Search Scoring \/ Ranking<\/strong>: Add frequency-based scoring using <code>$keywords<\/code> or <code>$index[word]<\/code> counts.<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\">This documentation should allow another programmer to <strong>understand, maintain, and extend the indexing\/search system<\/strong>, including handling new articles, keywords, or advanced search features.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n","protected":false},"excerpt":{"rendered":"<p>Overview The system is a static inverted-index search engine for your legacy PHP articles. It reads .php files from the articles directory, extracts content, builds three JSON caches, and enables fast keyword or search-term lookups without querying a database. The three JSON outputs are: Workflow 1. File Discovery 2. Metadata Extraction 3. Content Extraction for &#8230; <a title=\"How the WoodCentral Archive section works\" class=\"read-more\" href=\"https:\/\/www.woodcentral.com\/-\/peter\/how-the-woodcentral-archive-section-works\/\" aria-label=\"Read more about How the WoodCentral Archive section works\">Read more<\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-1634","post","type-post","status-publish","format-standard","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/posts\/1634","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/comments?post=1634"}],"version-history":[{"count":0,"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/posts\/1634\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/media?parent=1634"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/categories?post=1634"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.woodcentral.com\/-\/peter\/wp-json\/wp\/v2\/tags?post=1634"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}