WoodCentral Forums

Est. 1998 — 27 years of woodworking knowledge

Server upgrades

Posts

Server upgrades

Edited #1

admin

I try to be transparent about the site, so I'm passing along the latest email from the company that provides the hardware for our site:

Hello,

As you may have noticed we've had growing performance/stability issues throughout this year. Between issues with the database servers going crazy, the non stop rampage of AI bots scraping, and overheating/throttling issues with some of our Ryzen's, 2025 has been a struggle at times.

Thankfully, with the help of magical internet money, we've spent about as much as a decent sized house in the midwest USA in new hardware. Everyone gets an upgrade!

Old Hardware Specifications:

AMD Ryzen 7900 (12 cores, 24 threads)
128GB RAM
2 x 7.68TB NVMe

New Hardware Specifications:

AMD EPYC 9654 (96 cores, 192 threads)
384GB RAM
4 x 7.68TB NVMe

A massive 8x increase in CPU threads, 3x in RAM, and 2x the storage and i/o! These upgrades will be completed with minimal disruption, requiring only a few minutes to reboot to apply the changes. No data loss concerns, no extended outage, no IP changes! I'm sure there's concerns that this just means we'll load the nodes 8x from here, but that's not the case. We've set much stricter limits (with some nodes already meeting our targets on these Epycs, meaning they will get no new signups), and we'll be making coding changes to outright block provisioning to servers already marked as "full".

Now, with Vegas already upgraded we've been able test how long it takes to complete nightly backups. We've been able to cut average run times from 6 hours down to 1 1/2 hours. Foreman willing, we'll be able to start taking 2 backups per day, and keeping anywhere from 60 to 90 days of backups at no additional charge.

Additionally, we'll be announcing some changes to plans in the coming weeks/months. With resource upgrades coming to some existing plans, as well as some new higher resource plans - albeit well balanced in terms of user counts for the hardware so we're not looking for 300 core CPUs in another 18 months.

We're actively racking & shipping equipment to all locations and expect to have everything done in the next 45 days. Here's the current status for each location:

Las Vegas - Fully upgraded as of the end of July.

Netherlands - Gear arrived earlier in the week. Our tech will be racking half of the nodes in today and all will be completed by next week.

UK - Gear was delivered to our offices in Las Vegas yesterday. We expect it to ship out tomorrow afternoon and be ready for migrations in the next 2 weeks, pending any delays in new rack provisioning by our datacenter.

New York - Gear is ready to be shipped from Vegas tomorrow as well. This one will be completed sometime in September as Telehouse must provision the rack, and Francisco will most likely take a trip to New York to handle the install himself.

Singapore - Hardware has been ordered and will be delivered to Las Vegas next week. We expect it to ship out for Singapore and be ready for migrations in the next 2-3 weeks - we've got international shipping windows to contend with and the datacenter has to provision our new rack.

We greatly appreciate your patience with us throughout the year as we've had a number of issues come to a head at the same time. We've been actively working on changes behind the scenes for a couple months but wanted to share an update with you as we're starting to see the finish line.


They lease space and data racks in the best data centers around the world and white-label their configured servers to whose who wish to resell as a "web hosting service." This is true of most hosting companies, even the big ones everyone recognizes. They don't own the data centers but just lease space and equipment in them. The data centers are owned by HUGE companies most people have never heard of. In our case, I white-labeled this as ABOL which provides hosting for WC. Click for details on the specs. If you understand them, you will be impressed. If you don't, it will put you to sleep reading them.

I haven't noticed any of the issues they describe. Their up-time has been better than the 99.9% advertised by most hosting companies. If there have been glitches, it was probably my fault because I don't know what I'm doing. Or uhm, let's just say I'm learning new stuff.  You know, trial and error. Yeah, that's it.

BTW, our server is located in the Netherlands (Luxembourg), which is scheduled for updates today and the coming week so there may be disruptions. Why Luxembourg? They are the most secure in the world in a country with the strictest privacy laws. And their data centers have huge internet pipes to the outside world with lots of redundancy--just in case. The fact the physical server may be half a world away from you isn't a big factor. The site uses a Content Delivery Network (CDN) that caches things at about 300 data centers located around the world. If you're in Japan, the page is probably being served to you from Tokyo, not Luxembourg. Nothing but the best for WC!

Now if I could just keep from breaking things. :(

Re: Server upgrades

#2

admin

It appears they did the hardware upgrades over the weekend. Notice any differences--good or bad?

Re: Server upgrades

#3

Sign-ins are very slow, like minutes.

Re: Server upgrades

#4

admin

@Dave Bair,

That should absolutely not be happening. I am somewhat obsessed with optimization and consider anything more than even a fraction of a second delay to be bad. 

I am on the fence about this upgrade. They certainly put a lot of money into it, and theoretically it should be a huge improvement. For example, the AMD EPYC 9654 processors are about $5K each, and they probably are replacing them in dozens if not hundreds of nodes.

I am experiencing occasional "sticky" pages, but I also upgraded my router and started using a much newer computer as my "daily driver" during the past week. To me, it seems to be coming from my router. My old one was only 100Mbsp, and this one is rated for 1Gbps and will deliver it, but it seems to lag at times. However, there are so many potential bottlenecks as data sometimes literally traverses the entire planet (and from space if using StarLink) that it is difficult to tell sometimes.

I have a blog on WC where I put stuff for future self-reference and for those who may be interested. I wrote up the following as I was investigating this.

https://www.woodcentral.com/-/peter/fast-internet-slow-web-a-guide-to-spotting-bottlenecks/

Re: Server upgrades

#5

admin

For reference, LIghthouse is a widely used measure of performance, accessibility, best practices, and SEO, and we generate perfect scores in all categories.

pagespeed_852.jpg

Test any page here:
https://pagespeed.web.dev/

Re: Server upgrades

#6

Sign-in was it's normal, speedy self today.

👍 This page answered my questions

Your vote helps other woodworkers quickly find the answers and techniques that actually work in the shop.