We value your privacy.
We use cookies to enhance your browsing experience, analyse site traffic, and personalise content.
By clicking “Accept all”, you consent to our use of cookies. You can also choose
“Essential only” or learn more.
As of today, WoodCentral's oldest archived messages (vintaqe 2001), on the Messages and Turning forums will begin to disappear, slowly but steadily as new messages accumulate on the live forums. You see, we're getting close to the server's limit for the number of files we are able to store, so we have to start freeing up "inodes" (google it if you want an eye-watering technical explanation) by drawing down our old files at least as fast as we add new ones. Failure to do this would really mess us up.
So, those of you who value archived messages may want to start reviewing some of the oldest ones before they vanish. I will be saving copies of the raw message text files, but they will not be displayed or searchable again unless some wealthy archaeologist wants to write a stand-alone application that will decode and display them as you see here. There are still about 900,000 messages to browse if you're so inclined.
As I understand it the issue is storage space. Likely only a small fraction of what is archived for a year is of lasting value. Is there a means of deleting the chaff and keeping the kernels?
Selecting the kernel from the chaff, sheep from goats, etc., would require someone to sit and review every post and make that call. I know I don't have the time; I doubt Ellis does; not sure who would.
perhaps certain filters could be applied to guess which threads were the quality ones. For example:
-- most responses
-- highest wordcount
-- contain pictures
-- number of views
number of views might be a particularly good one, if that info was collected back then. posts with exceptionally high view counts might be the ones that people are still finding with search engines long after the conversation was over.
The imperfection of this idea will be obvious to all. The thought is that something could be better than nothing. Suppose it would be technically practical to parse out a month of posts to 12 people who by their judgement would imperfectly eliminate 90% of the posts(or any fraction for this example). Would what remained be more or less useful than tossing them all?
Another proposal- delete all NM posts, or keep all PIC posts. Again, with the focus on the concept that anything is better than nothing.
I've got essentially unlimited storage space. It's the inodes that are the limiting factor, and those are directly related to the number of files stored.
if you're willing to go through and hand-delete 750 files per week for me, we'll keep the kernels. I do regret that we have not somehow tagged great threads over the years, but we haven't. I've talked with my developer in Germany about utilizing a utility of the software that would identify the good stuff, but we'd still need an editor to keep track of it. Would that I had the time and temerity.
Does the software have any ability to combine older messages into single, static files? One file per a months worth of postings? It seems to be a failing of the software to use this many individual files. It has always been my understanding that even if you had a dedicated server without an artificial million inode limit, it can still affect performance to have that many files.
This group never fails to amaze me. My notice on the other main boards has evoked barely a murmur, but here it has turned into a vast and complicated discussion. Good on ya.
I'm going to free-associate a bit here, in response to your constructive comments. First, let me say that David echoed my personal sentiments when he said that the archives are not the essence of our association here. For me, it is the timeliness of information, not the historical record of it. I know from my magazine days, though, that a lot of people (dare I say half?) are what I'll call archive hounds. They enjoy browsing through old discussions the way a prospector pans for gold. It takes all kinds, which is why I've tried over the years to maintain at least a tattered collection of archived messages. The voids and holidays that you see in the archives are the result of my meddling around several years ago, when I managed to move piles of messages from one bucket to another in order to try to preserve a semblance of continuity, as our software sometimes loses it when late messages are posted to an earlier thread. That is a function of a policy I put in place about 10 years ago, which is to archive all messages earlier than the most recent 5000 on each of our main boards. Depending on traffic, that could mean anywhere from a couple months to many months, meaning that later messages are no longer grouped in the same time-frame bucket. I'm on conceptually thin ice here, so I'll leave it at that.
I'm most amused (in the good sense) though about the ideas about how to pare down the archives without losing the good stuff. The idea of tossing *NM* and OT messages, and preserving ones with associated photos is indeed doable, but it will require me to hire my Perl guru in Hamburg, Germany to write a pretty major piece of scripting and then run it on all the archives, and then reindex everything again. At €95.00 per hour, that starts to add up to real money, so I need to invest carefully. But, I believe it could be done. Whether we could find 750 such files per week in perpetuity is doubtful, so this would likely be a flash in the pan. If we wanted to, we could also just compile a list of the people whose posts are customarily most worthy of keeping, but that would be silly and unfair.
The best way to sort out the good stuff is manually, and there are at least two large obstacles to that. One is that it requires a commitment of time by one or more people, which most certainly would include me. The other is that if we enlist a team of volunteers for this task, I still couldn't grant server access to them, so all they could do would be to note the message number and then pass it along to me for deletion. I would have to manually open each folder -- messages are grouped on the server in folders of 1,000 messages -- and find and manually remove those designated for deletion. Another glitch there is that we'd have to lock the boards for several hours (in the middle of the night in North America, to minimize inconvenience for visitors) and run a custom indexing script on all the messages -- on a weekly basis. More aggravation and maintenance expense.
What I really would like to do is to hire a combination Perl/PHP guru to convert all our flat message files to a database, and then write a script that would compile and display them much like you see them here. I am pretty suire this is possible, but I'm thinking it would be in the mid-five-figures range to do it. ya gotta wonder what the purpose of all this work might be.
Maybe there are other possible courses of action that would preserve past gems, such as moving them to separate message collections on our present server, so that when the weekly delete process occurs, it wouldn't grab those with the others. I can look into that with our guru and get back to you.
I'm very gratified to have all your good ideas and participation in this discussion. It bespeaks your real concern and respect for what we've accomplished here. I share your concerns and will continue to try to find some way to preserve the best of WoodCentral. Time and money are the big constraints.
The only reason I've hung onto this software is because, in itws retro way, it is the most readable option. I could have bought into vBulletin or PHP-BB or some other vanilla PHP/MYSQL messageboard package years ago and reaped the benefits of quick, solid searches, as well as the features and advantages of those packages (who's online, pm'ing, avatars, linked personal galleries, etc.) Others have, and many people prefer that format for reasons unclear to me. I have run several plebiscites over the years to gauge our visitors' preferences, and they've consistently come down on the side of our existing software, old and flawed as it is.
Our forums are run by a suite of Perl scripts that group and display them in whatever way we set the configuration parameters. The texts of the messages, plus metadata such as (encrypted) passwords, e-mail addresses and usernames, along with timestamps and ID numbers, are stored in flat text files that are called by the scripts and formatted for display. Conceivably, we could fold them all into large files -- through some coding sleight-of-hand -- but it would destroy the integrity of the Berkeley database structure and render them unusable.
Haven't totally thought this one all the way through yet, but is there some way the messages could be exported to a common file format, like for instance a csv text file? Then the output of the csv files could then be stored in a winzip type archive? Perhaps, a month at a time?
A csv file format could then be imported by other programs such as Excel, and (I'm not a database person), but I would think a database could then also import them as well.
I guess nothing could be less usable than deletion, although, for practical purposes, they'd be really about the same. Say for example that you searched for any search term, and hundreds, if not all, of these mega files showed up in the results. Each one would weigh in at about 1 Mb, so depending on your access mode, it could be a long download. Then, once you get it. you would have a chronological listing of posts, not threaded or formatted, with no associated photos.
I liked Tim's idea of converting them to, say, .csv files, but the integrity of the threads would still be lost unless you could somehow preserve the child-parent relationship by some sorting function.
By way of illustration, here is how just five messages would appear:
Just delete all threads about bevel up planes. The topic resurfaces like dandelions, and there's never any new information, just a spew of re-hashed opinions.
That should take out a good size chunk of the backlog.
The problem you have here is that the number of files allowed in the filesystem you use (some Unix or Linux type I gather, given that you mention inodes) is fixed at creation, and can not be changed afterwards. However, not all filesystems in the Linux/Unix world have that limitation. In particular, ZFS allows 2**48 entries in a directory, I'd say more than you will ever need
I know I'm not the one who will doing the work, but the armchair quarterback here says that all you need is to do is backup your data, erase the current filesystem, create a ZFS one (Linux support is native now), and restore the data onto the new partition.
Of course, things are never that easy in the real world...
Our server can be set up to accommodate more files, but our ISP has a policy limiting the number, presumably based on their daily backup conveniences.
When I signed on with this provider, there was no set limit, but about a year ago they changed their policy without notice. I actually shopped other ISPs after that, but I never found someone who would rent me that kind of space without a major bump in fees.
I make it a point to be marginally conversant with all this stuff. Beyond that, I realize I will never be a guru. Gotta hire the folks to do the work.
Is it possible to store off a whole chunk of time as an integrated unit? For example, to move 2001 completely off line in such a way that it can viewed as messages are now? Maybe volunteers will show up to find and preserve the nuggets - with whatever software?