Tech CAguns.net Engineering

[ikeo1]

It's been another whirlwind week at CAguns.net, and the landscape has shifted considerably. With Calguns back in operation, we've noticed a significant change in our site's metrics—sign-ups and new posts have slowed, a testament to the strong loyalty members hold for Calguns.

Despite this shift, the fluctuation provided us an opportunity to refine and improve. We've addressed several issues that emerged during the surge in traffic, applying throttles—common in cloud-native applications—to maintain service quality and speed.

Now, with these challenges managed and throttles updated, we're poised to focus on growth once more. Changes in storage and bandwidth management, supported by new sponsorships, have opened new avenues for us.

However, the competition remains formidable. Calguns, despite its UX challenges, maintains an active marketplace. Our strategy will be to continue refining our features to enhance user convenience. The introduction of Sub-Regions has already shown promise, with quicker sales transactions by connecting buyers and sellers more efficiently within their local areas.

Thanks to the support from our patrons and advertisers, we've reached new milestones in service provision, positioning us to sustain site operations and further development. Another area we'll be working on is our documentation to clean things up and make an easier path to onboarding.

Looking forward, while the immediate growth spurt has tempered, the foundations laid by our community's support are strong. CAguns.net remains well-prepared for future opportunities. We're grateful for the journey thus far and eager to embrace what comes next, confident in our platform and the community that makes it thrive.

 

[W.H.O.]

That’s not a problem right now. We’re not serving a ton of data yet so it’s ok. Eventually what we’ll have is different layers. The free tier can store 1 cover photo and link out while contributors can upload multiple images directly. Since storage and bandwidth costs are a consideration I think this would be the most fair approach. You can still submit images but not at my expense. Then offer a filter based on that. This isn’t til later though and I will likely grandfather existing users of the platform once it happens with a limit of 3 images.

As far as image hosting goes, it would probably be best to have a dedicated image host for the forum instead of utilizing 3rd party uploads to mitigate breadcrumbs & utilize auto deletion of images after a certain time period for the sake of space saving. (I.e. once image reaches 30 days uploaded, uploader gets prompt asking if they want to keep it uploaded for another month; no response in 72hr returns server request for auto delete...etc).

 

[ikeo1]

As far as image hosting goes, it would probably be best to have a dedicated image host for the forum instead of utilizing 3rd party uploads to mitigate breadcrumbs & utilize auto deletion of images after a certain time period for the sake of space saving. (I.e. once image reaches 30 days uploaded, uploader gets prompt asking if they want to keep it uploaded for another month; no response in 72hr returns server request for auto delete...etc).

We use a dedicated CDN host for images using object storage similar to AWS S3 since its cheap storage, but don't auto-delete as that would cause a lot of broken images. They are deleted if the user deletes them though. We utilize caching for 30 days.

 

[W.H.O.]

We use a dedicated CDN host for images using object storage similar to AWS S3 since its cheap storage, but don't auto-delete as that would cause a lot of broken images. They are deleted if the user deletes them though. We utilize caching for 30 days.

The auto delete was in reference to user uploaded images specifically, such as for classified ads. Totally understand what you mean though. Imagining the site breaking every month if you didn't confirm every image embed sounds like actual hell. Lmao

 

[GeeQ87]

New to this site, not from Reddit. So far, this is the easiest to navigate and use. Fantastic work!

 

[ikeo1]

New to this site, not from Reddit. So far, this is the easiest to navigate and use. Fantastic work!

Thanks! Welcome to the site!

 

[ikeo1]

I thought I'd follow up here to show the growth we've had recently here at CAguns.net. Those of you who've followed along early saw our big growth spurt early on during the original Calguns migration. At that time, we were still fresh and didn't have a lot of content. This time, we have almost 2.5x the number of users from less than 1000 to now just over 3800. In the last 3 weeks we've grown 900 users and a few hundred listings.

Not much has changed on the backend since the first spurt forced us to make change some decisions early on with our hosting provider and our tech stack. Three months later with and a larger influx of people we're managing the much better. There's no performance drop anywhere and all parts of the system are behaving well. The database and webserver are less than 20% utilization at 95% percentile with a mean of 7% utilization over the last 7 days. The database works less than the webservers which is a good thing. We're not even close to pushing the servers and at anytime, we can increase the system specs (CPU/Memory) by rebooting the box. There's no need to migrate.

Over the last 7 days, cache hit rate is 87%+. Given the same number of users from the previous period April-May. The amount of content is more than last time, from 5.7 million requests up to 8.1 million and growing. The amount of data served shows the increase of an extra 50% 153 GB vs 105 GB. How this compares to Calguns, is the gold plan for vbulletin was $60/mo for up to 200GB. Considering we use some other service our current run rate is ~$100/mo not including software licenses. Fortunately, with limited advertising and memberships we're on our way to breaking even and turning a small profit. We currently only have 1 direct 2A sponsor @T3 Pros , which we really appreciate.



The curve for this level of growth is a lot nicer where once CGN came back online our userbase dropped. The same thing is likely to happen when they come back online again, but this time more people are familiar with our features. We'll continue to do our best to add more content to the site and allowing users to contribute content.

There's a lot of people who say there's still a small amount of users, but I'm optimistic what we offer here is a solid experience at no cost, with a growing community. You can contribute if you want to in several ways, by sharing, asking and answering questions or via donation, all while the service is FREE to use thanks to early supporters. I'd like to thank the CAguns Supporters who allowed us the breathing room to continue to build freely while we build up our content.

As a side note, this is the first time we've nearly hit 1000 active users daily, considering there's 3800+ members that means over 25% of the members are active daily! It's a really strong sign for us that we're doing the right thing. Please keep spreading the word.

From time to time we like to share these metrics.

We do have some upgrades planned in the near future, so be on the look out for those.




Cheers!

 

[yatzooa]

It's probably impossible to do this overlay, but It'd be funny to get a graph of unique visitors or new visitors vs calguns.net website availability LOL

 

[ikeo1]

I wanted to provide a small update here to show how far optimizations can go. We've increased our active user base 4x-5x and continues to increase with the recent downtime. I follow metrics a lot and I'm fairly passionate about performance and efficiency.

The metrics below show the power of having different layers of protection and N tier architecture as noted in earlier posts. The benefits are showing now as our traffic doubled and continues to do so with a minimum impact on the CPU/Memory utilization. We rarely go above 20% utilization and when adding CPU/Memory only requires a reboot. Database utilization rarely goes above 10% and webservers barely crack 20% on p95. If using mean, the avg is lower at around 10%, but with 95th percentile we can see a clear picture of most spikes in the system.

Strategic use of our location and network. Any pings from around the state to our network should be less than 10ms. You gamers know what I'm talking about.

You'll be able to notice the light blue line is the Database has low CPU but higher memory as we cache most of the requests. The webserver has a little bit higher CPU mainly responding to connection management but lower memory so we'll be making some optimizations there on the next round.

We use ARM processors which are very efficient and specialized for the type of work that we do.



The system architecture is holding up well serving close to 10 million requests in 30 days and can handle more quite easily. You can see in 4 days we serve approximately 8-10GB of traffic daily. We don't have as many users as Calguns but we're not far off either in terms of active members. The current setup can scale another 3-5x from its current point so we'll start our own migration soon to upgraded software and hardware.

We'll be leveling up since it seems like there's real traction and we can move on to the next stage of growth.



With the recent growth in mind, I'll be performing an update but wanted to show how much headroom there is before its actually necessary. The recent donations and subscriber growth has allowed us the flexibility to procure a staging environment where we'll be performing an upgrade to the latest xenforo 2.3 branch. We'll be testing heavily while we get ready for it. Technically its not necessary, but there are a number of enhancements that will improve our overall performance and ability to add new features. This should all be mostly transparent. Our focus will shift from what CG was doing to focus on our own path.

Thank you again to all the donators and subscribers and our new vendors. They give us the headroom to invest in the future of this platform and with some cool features on the roadmap once we settle into the new version.

 

[yatzooa]

How's the search function latency? Are you seeing any slow/gradual increase in latency as the site gets more and more posts?

 

[ikeo1]

How's the search function latency? Are you seeing any slow/gradual increase in latency as the site gets more and more posts?

No, we have plenty of room for growth. Most of the search is text and not blobs so because I limit the number of chars in the description its pretty scalable.

On the next upgrade, we'll be moving up to 8, then reindex the entire site while its still small.

 

[SiRMarlon]

This is pretty awesome to see. Thanks for the transparency for us geeks who actually enjoy like this! I am actually in the middle of doing a lift and shift for some of our cloud based software. Going from a managed service provider to Azure so it's going to be pretty fun!

 

[ikeo1]

This is pretty awesome to see. Thanks for the transparency for us geeks who actually enjoy like this! I am actually in the middle of doing a lift and shift for some of our cloud based software. Going from a managed service provider to Azure so it's going to be pretty fun!

No problem. For me, this is like some notes while building up the site for posterity. It was originally just pet project to see if I could create a cloud native version of forum software but its turned into so much more.

I wouldn't have imagined the site would be where its at now.

Good luck on the migration. Azure is pretty nifty. All the telemetry data is going to be fun stuff!

 

[SiRMarlon]

Good luck on the migration. Azure is pretty nifty. All the telemetry data is going to be fun stuff!

Thanks, we already got some of our services stood up and I am just blown away at how much faster we are delivering content (I am talking about big CAD files) to our users. We easily got a 20% bump in performance making the switch from our previous provider to Azure. The funny part to all of this is that their datacenter is located in Las Vegas. We are placing all of our services in Azure East due to it being centrally located from all of our Offices. We are in Huntington Beach, Central Coast, UK, And Germany, and we haven't even begun to really make optimizations yet. We are just in the begging of our testing phase.

 

[ikeo1]

Thanks, we already got some of our services stood up and I am just blown away at how much faster we are delivering content (I am talking about big CAD files) to our users. We easily got a 20% bump in performance making the switch from our previous provider to Azure. The funny part to all of this is that their datacenter is located in Las Vegas. We are placing all of our services in Azure East due to it being centrally located from all of our Offices. We are in Huntington Beach, Central Coast, UK, And Germany, and we haven't even begun to really make optimizations yet. We are just in the begging of our testing phase.

The one big thing is research all the Azure limitations and throttling there are. Hella annoying but all the limits are there for "reasons". There's probably limits on everything and depending on your application, you'll run into all kinds of throttles that you didn't have in a managed environment.

We use that same concept here. Tons of limits 20% of the people might not like it but service will stay up and within bounds

 

[DrJonesUSA]

The past couple of weeks have been a whirlwind since we launched our website. Originally operating under a different domain, we made the transition to our current domain just 10 days ago. I'd like to share some insights and the motivations behind dedicating my efforts to this platform.

Like many of you, I have a deep appreciation for Calguns, despite its technological shortcomings. Recognizing the need for a fresh and improved platform, I started creating our site. The data from our brief operation suggests a strong demand for a modernized solution. Since migrating to an optimized platform a week ago, we've observed a consistent increase in traffic, indicating genuine interest.

An important aspect I want to highlight in this post is our site's efficiency, with 63% of its content being cached. This approach significantly enhances performance and user experience by reducing the direct load on our database. Our experiences with CG have informed our technical strategies, leading us to alternative forum software and architectural designs aimed at avoiding similar pitfalls.

View attachment 297
Our infrastructure employs multiple caching layers to ensure smooth operation. We utilize Redis for session storage and Memcached for public page data, supported by CDN edge caching and extended browser cache policies for static content. Further database query caching minimizes direct SQL calls.

For content delivery, we rely on object storage and CDN rather than local storage for images, allowing for faster page loads with subsequent image loading. We've limited image sizes to 1280x960 pixels, which suits most devices, and are considering a shift to the .webp format for enhanced compression without sacrificing quality.

Our backend is designed for scalability, prepared to dynamically adjust storage, memory, CPU, and balance loads according to demand. Although it's early days, such foresight in planning speaks volumes about our commitment to growth and efficiency.

I extend my gratitude to the r/CAguns Reddit community for their invaluable feedback, encouragement, and support. The future of this platform lies in the hands of the California 2A community, and I am committed to leveraging the latest technology to serve our community without the constraints of major tech oversight.

Please let me know in the poll if this type of content is something that you would like to see and hear from. If so I'll add different aspects over time as the site develops.

uh.....can you please Explain Like I'm a Golden Retriever?
LOL....... just kidding.....
Seriously; this is a FAST site!!!
I'd definitely noticed how pokey CalGuns had gotten.

FWIW: I plan on hanging out a bit at both sites, once calguns is back up, but I'm happy to have a place here too.

 

[ikeo1]

We use ARM processors which are very efficient and specialized for the type of work that we do.

View attachment 7428

The system architecture is holding up well serving close to 10 million requests in 30 days and can handle more quite easily.

View attachment 7427

Checking in after we've stabilized from the recent influx of new users. We've grown our users to nearly 4800 members (+1000) and traffic about +53% since the last time this was taken a couple weeks ago. Our traffic increased from ~9M requests to 14.48M. A jump of just under 5 million all with very little if any impact on performance. The number of unique visitors continues to climb but the growth rate has slowed a bit since CGN has come online.




You can see the spikes where we did updates, which most people didn't notice, except for that last spike. Periodically we look at the growth rate and compare the utilization to see how the optimizations are working and to keep a rolling history when more traffic comes. You can see the drop in traffic once CGN comes online and flattens out as users go back and forth between the two sites.

One thing we always keep tabs on is our performance because its an interesting challenge to build up systems like this. At nearly 300GB we're already serving 1/3rd the size of CGN's active user base or closer to 1/2 given the performance. I'm pretty sure we can continue to scale without any additional changes to the infrastructure, but we are moving soon to ElasticSearch 8. We've upgraded our PHP version and Database version to the latest and the last one is to migrate the search off. This is thanks again to our CAGuns.net supporters who are contributing so that we can separate the service and continue to make it more reliable and fast.

We've removed some things from the database and will continue to update the site periodically, but for now, the issue is more moderation than performance. Moderation seems more difficult than performance tuning for us.



Stay tuned for more. We probably won't post again until there's a significant increase since the system is behaving.

 

[foothiller]

Excellent site. Function, features, and performance. Thank you!

 

[ikeo1]

It’s always fun to look back at these statistics and see just how much we’ve grown. Over the past six months, we’ve seen a 300%+ increase across several key metrics. The most recent dip was due to some filtering we applied, so while there’s a visible drop, the overall trend still shows a significant increase in activity.

We’ve now completed the migration, updated all software packages, moved to an enterprise-grade operating system, and further tightened our security. It’s exciting to see total requests hit 50 million, with data usage approaching a terabyte per month.



CPU & Memory usage is higher now but according to GPT which is helping me analyze the results the new system is more stable and consistent. There were some issues with some errors but I think that's gone down. We have plenty of headroom for the time being and will continue to monitor.




Some people might wonder why I’ve been working to monetize the site, and here’s the reason why: we’ve seen a 3x increase in traffic and usage, but only a very minimal increase in donors. As a result, I’ve had to lower some things down and may need to increase ad placements around the perimeter of the site.

If you become a donor, the ads will be limited to local advertisers/vendors we want to promote and encourage to participate in this space. Vendors also help support the site, and our goal is to keep everything self-sustaining as the site and community continue to grow.

We also maintain a staging area for testing, which allows us to roll out updates cleanly and with minimal disruption. I hope we can continue growing the site and offering more features. It’s been a lot of fun, but I’ll admit the marketing side of things can be tiring, it’s not my strongest skill. From a technical perspective, this is a great hobby, but it’s becoming pretty costly. Some of the recent upgrades were made to improve stability and efficiency, ultimately helping reduce costs. I also cleaned up and consolidated servers, making it much easier to spin up new instances when needed.

Now that the major upgrades are done, this marks over a year of patiently and consistently building and scaling alongside the community. Going forward, updates should be easier and more streamlined. There’s still work to be done, but it’s far less than in the past year.

Finishing the migration is a huge milestone, and I’m glad to have reached it. It puts the community on solid technical footing for everything ahead.

 

[vplaza96]

I really appreciate the work you do for this site, both technical and content/moderation wise.

This site is fast, easy to navigate, and I appreciate that it's perfectly accessible from mobile devices.

Keep up the great work!