Announcement Migration Plan

[incognetpanda]

Here you go. No download needed.

How To: - Use the CAguns.net Mobile App! (Progressive Web App)

We’re always looking for feedback from early adopters. So anyone who tries it and provides feedback will get a badge for their trouble. This badge will only be given out for early adopters. This will be closed once we've received enough feedback. Please include a screen shot of the app for...

caguns.net

I am on android nowadays but I figured it out thanks to you so thank you

 

[NorCalBusa]

Yeah. that window closed a long time ago. It was good! 49'ers finally had good defense and strong offense!

Guerendo got hurt, Niners had to sign a 4th string running back.

Not too bad it seems, might miss just 1-2 games. Seems the Trainers, in the last few years, have figured out the best recovery process is not speed.

 

[NorCalBusa]

I am on android nowadays but I figured it out thanks to you so thank you

OT- have you dug into the iPhone-Android texting FBI concern? https://www.forbes.com/sites/zakdof...ange-whatsapp-facebook-messenger-signal-apps/

 

[incognetpanda]

OT- have you dug into the iPhone-Android texting FBI concern? https://www.forbes.com/sites/zakdof...ange-whatsapp-facebook-messenger-signal-apps/

not a huge deal for me worst case they breach my "privacy" which is a straight up illusion on the internet to anybody who knows there stuff. and then there's those who can use it while actually being anonymous and those guys are freaky individuals

 

[ikeo1]

Just an Update on this.. We're moving full steam ahead and will be doing a blue & green migration . For those of you that don't know what that is, it means we'll be doing the upgrade which takes a little longer but is easier to roll back in case of an issue. I've been working on setting up a new system that should provide more stability and performance, changing the underlying os and improving the CPU to the latest ARM processors. I'm currently in the testing phase, but wanted to inform you that this may take a few hours to complete. I'll provide the expected update day. In the meantime, just be prepared. I'll post again as it gets closer. For now I'm preparing. It's looking like in the next 2 weeks I'll be ready.

Thanks for all the patience! Lots of things on the way. I'll space things out a bit more in the future.

Cheers!

 

[ikeo1]

Took a little longer than expected but this milestone is now complete. We'll continue to forge ahead this time modifying the system to be more tailored for our use case and polishing up rough edges. The site is maturing nicely and we're making consistent progress as we iterate.

The next step from here is a migration to a new server set which will come a bit later which will enhance our security and OS stability. We overhauled the front end but next up is the backend platform. We'll be migrating from Ubuntu over to Alma Linux which is more enterprise grade and will future proof the base layer. This also allows us to update our deployments and migrate to Docker containers which I've already been testing which will help even more with consistency.

This site is totally over engineered for what it is. But it's fun. I'm also building an AI support agent that will answer questions about the site and do some light tasks. Testing technology I'm building for my own education leveraging LocalLLM's and other cutting edge tech to see if its useful or if its hype.

 

[ikeo1]

Fortunately the updates are a bit faster this time. We've increased the size of the Database version, OS kernel, and web server with pretty minimal downtime. We've been growing pretty quick and I wanted to make sure if there were any issues I had time to work on it.

Things are running smoothly here and you might notice a little performance boost on page loads. There's still more work to be done here as the site continues to grow. It's been fun working on the site and taking it to new levels.

 

[Striker777]

Maintenance Complete..

We had a brief hiccup due to a networking issue. Sometimes these things happen when things get a little loose. Additional security was added which blocked some calls. Now that we're more popular, we're a bigger target and need to actively defend and tighten down things.

To do:

• Create update page to point to discord server for updates.
• Create non-expiring Discord Link
• Updates will go to the Discord channel in the future.

Tighter Security includes:

• External & Internal
• An example recently we had users using temporary email service to register an account
• If an email is changed, you cannot post, and must reverify
• We have additional measures to counter scammers

On that note,

We cannot prevent Low Ballers or trade offers... Please be respectful to the hustle... If you don't like a DM, you may use the "Leave" button to ignore the conversation.

View attachment 8504

Yes... the leave button well that's better than nothing does the leave button give the prospective buyer a message that you no longer want to deal with them (sort of speak) I hope!

 

[raellic]

I have a side business as a systems integrator/programmer. I want to say that this forum is highly technically advanced and your work on it is greatly appreciated. One suggestion for @ikeo1 -- have you considered bypassing the database entirely and creating static page components that do not require any queries? I did this with my own BBS software that doesn't have a database at all, and I got it down around 25 ms to load individual threads. Would be interesting to compare notes one of these days.

 

[doser]

I have a side business as a systems integrator/programmer. I want to say that this forum is highly technically advanced and your work on it is greatly appreciated. One suggestion for @ikeo1 -- have you considered bypassing the database entirely and creating static page components that do not require any queries? I did this with my own BBS software that doesn't have a database at all, and I got it down around 25 ms to load individual threads. Would be interesting to compare notes one of these days.

Using a database allows for a lot basic features. The site itself is served mainly via cache so the hit on the DB is minimal. There's also several layers of caching in the layout and when we've tested. Initial response is very fast less than 10 ms if you're in NorCal but payload is 500ms for 3-5mb of data when already cached. SoCal is only minimally more. Given the customization, the platform is scalable to a million users and based on a more traditional software stack. I help from time to time, but there's also a few dev's who are involved with the custom work that needs to get done if we don't do it in house.

 

[ikeo1]

We'll be moving on to the next phase of the upgrades as the site continues to grow. The next phase will require a little bit of downtime while we migrate databases over to a more stable and enterprise ready OS flavor. We'll be moving from Ubuntu to Alma Linux 9, which will make the upgrade path very predictable and low maintenance. The next phase after that will be to migrate off the main web server and do the same type of OS migration. We've tested extensively on texanarmory.com and armoryhq.com and the management and overhead is reduced significantly with some small performance improvements.

Don't worry, we do blue/green deployment, meaning both databases will be online so reversal is easy. Shout out to our CAguns Supporters and Vendors for helping us fund the transition. We spin up multiple environments so we can roll back if necessary.

As we approach 20k, we will have to make some changes to the free tier as our costs have significantly risen over the last few months. We're growing faster and faster thanks to word of mouth growth. The changes were all discussed last year so the main reduction will be with the number of listings available going down from 5 listing to 3 and a new subscription will be introduced at a lower rate.

Again just to remind people 20k users x 5 listings is 100,000 listings * 3 images (300,000). Advertisers and CAguns Supporters are currently supporting the free users. Our number of items listed and sold are continually increasing as more and more people sign on our supporters to free members ratio is widening.

We'll be sponsoring more matches to get the word out.

Everything is growing and we're adjusting to it. Thank you for all of your support!

 

[Buck Liggett]

I found this site do to Calguns demise.

 

[ikeo1]

We'll be migrating the database within the next few days. I'm finalizing testing making sure everything is in working order and all update scripts are up to date. I'll post a notice when we're ready to go.

Thanks everyone for pushing the servers hard! Once we do this move, we shouldn't have to migrate databases for at least a couple years. I still have one more move on the app server. Once that's done, this thread will be complete.

 

[ikeo1]

Database migration complete. Total downtime was approximately ~45 minutes. Mostly due to sanity checking. The actual migration was completed in < 10 minutes. So far so good, please let me know if you see anything else.

Now there's only one more migration left with the webserver. Then this thread will be ready to close...

Some notable changes:

Application

• Added a section for vendors to showcase deals to the community - Vendor Row
• Moved number of listings to 3 from 5 due to consistent growth

System level

• Maintenance tasks should be faster and less frequent
• Optimized Alma Linux OS
• Migrate from ubuntu to Alma Linux 9.x
• Increased IOPs and Throughput on DB disks
• Moved DB storage to dedicated volume
• Upgraded to a dedicated monitoring system
• Upgraded DB version

 

[wtf_calguns_migration]

You were running data on instance?
Yeah, as your platform provider makes block storage tiers available cranking iops up is trivial once it's decoupled from the db.

It also makes availability less of an issue because minor or sec updates to the db host OS or db engine can be tested on a data snap. Tended to be mandatory around 1999 when I was managing Oracle clusters as master lag was a thing.

Thruput usually increases by vertically scaling the db instances to get more performant virtual nics and interlink bandwidth.

Shifting to a a Rhel kernel depends on the db engine's affinity and specific query hygiene and velocity.

No cache layer makes all of this brute force.

 

[ikeo1]

Shifting to a a Rhel kernel depends on the db engine's affinity and specific query hygiene and velocity.

No cache layer makes all of this brute force.

There’s more than that.

• RHEL kernel is more stable for server systems
• Support updates are less frequent
• Support duration is longer 5 year Alma is 10
• Ubuntu release cycle is 2 years
• Alma is 3 years per major release
• RHEL is also optimized

I originally chose Ubuntu because it was fast and easy to develop on but the updates were too frequent and had one instance of an update taking down the system.

Now the OS is stripped with a lot less on it since we’ve been running for a year+ on this vm.

When it comes to caching, we have ~5 layers of caching since the beginning. Redis, Memcache, OpCache , disk Cache, and Edge Caching.

We've moved the service to a fully scalable architecture. I had it on the boot volume at first and did regular backups but with the growth of the site wanted the flexibility to tune and expand as needed.

In this case it’s not tied to a specific instance type. So scalability is very easy. Throughput/IOPs can be scaled by increasing disk size, two commands or independently via button. There’s no need to change instance types on this setup. Everything is scalable independently.

We’re serving upwards of 3.5 million page views a month now, so it’s quite a lot. I wanted to get ahead of the growth since it hasn't slowed down

You were running data on instance?

No, it's purely cloud based on a differenct provider. Instance types are more for AWS.

It's kinda fun having your own service like the one you build for other people. Also updated was a observability service for logs, infra, and app metrics.

 

[NorCalBusa]

There’s more than that.

• RHEL kernel is more stable for server systems
• Support updates are less frequent
• Support duration is longer 5 year Alma is 10
• Ubuntu release cycle is 2 years
• Alma is 3 years per major release
• RHEL is also optimized

I originally chose Ubuntu because it was fast and easy to develop on but the updates were too frequent and had one instance of an update taking down the system.

Now the OS is stripped with a lot less on it since we’ve been running for a year+ on this vm.

When it comes to caching, we have ~5 layers of caching since the beginning. Redis, Memcache, OpCache , disk Cache, and Edge Caching.

We've moved the service to a fully scalable architecture. I had it on the boot volume at first and did regular backups but with the growth of the site wanted the flexibility to tune and expand as needed.

In this case it’s not tied to a specific instance type. So scalability is very easy. Throughput/IOPs can be scaled by increasing disk size, two commands or independently via button. There’s no need to change instance types on this setup. Everything is scalable independently.

We’re serving upwards of 3.5 million page views a month now, so it’s quite a lot. I wanted to get ahead of the growth since it hasn't slowed down


No, it's purely cloud based on a differenct provider. Instance types are more for AWS.

It's kinda fun having your own service like the one you build for other people. Also updated was a observability service for logs, infra, and app metrics.

View attachment 64778

Man- I was just about to type all that and you beat me to it. Not...

 

[ikeo1]

Man- I was just about to type all that and you beat me to it. Not...

I tried to give the TLDR version.

I spent a lot of time with making sure upgrades go smoothly and everything is heavily tested before moving. I enjoy the system aspect of it and more specifically the reliability & performance aspect of the site. I nerd out on it like my pistols and ammo loads. It's become clear to me, that this site is a thing now, so I want to treat it with care.

 

[ikeo1]

Just some notes for working on this for the last few days.

• Overhauled base layer infrastructure to use docker containers for the web app
• Cleaned up and organized script base and put everything in a git repo
• Much cleaner for CI/CD pipelines and automating releases and rollbacks
• New backup scheme
• New Log rotation scheme
• New System Alerting & Monitoring system
• Tighter restrictions on system access
• Tear down old systems
• All packages updated with latest security and performance packages
  • PHP 8.3.21, NGINX 1.28, etc
  • Updating is significantly easier now via docker containers and automated build process

Not much to do after this.

One thing i found when I was digging around was that Amazonbot was really driving up the site cost. Cache rate went down from ~88% down to 66% and lower. There was a sharp increase in total data served ~100GB+ in a month with a big drop in cache hit. This led me to investigate and found that Amazonbot and a few other crawlers were eating up a bit of bandwidth and skewing the metrics. The additional monitoring, found some other areas I could tighten up.

I've made some changes and 24 hour hit rate is back up to ~88% so hopefully that'll bring down costs and limit the number of users. You can see this in the reduction of guests. We only want the good bots that will bring people here. This is a big problem for any independent site. The large tech companies are indexing anything and everything to build their models. They consume a lot of data and constantly refresh. The number of AI bots has grown significantly. The drop in the uniques towards the end were recent changes I've made to reduce bots crawling the system.

 

[ikeo1]

Just finished up migrating web servers and testing live migration.

System resizing takes ~5 minutes to resize and add more memory, cpu, and bandwidth. The service should be good to go. Old server stopped so some people might have an issue, that will just be a hour tops given our infra for everything to resolve.