Bugs 504, woff2 and a helping of brotli... plus zstd

[command_liner]

Almost every site I check has several odd configuration issues. By "odd configuration issues" I mean bad compression and
generally non-optimized content delivery. The calguns.net 504 issues are just one example, and they have ongoing problems
with invalid HTML pointing to non-existant objects. Many, many sites have similar issues.

Every copy of the Chrome browser now contains a good debugging tool to make spotting gross errors easy. The
view/development_tools tab has a network diagnostic feature. Just turn off cache and refresh the page. Every
page object will be analyzed in detail as it is downloaded. The size, timing, compression, sequence and relation to
every other object on the page will be displayed. It take about 3 seconds to set up such an analysis and start it running.

As it is running, set the sort to put the largest object (in delivered bytes) to be shown first. This generates a Pareto
of problems: the biggest objects are the slowest delivered. Simply optimizing the top 20 objects in this list is almost
always a sure-fire way to reduce time-to-usability of the delivered page. Is it work cutting the page load time in half?
Almost every user will say "YES!".

First thing to check is use of gzip. This compression format was great in 1995, but is obsolete. Any page containing
gzip should be considered invalid. There is lots of gzip here. OK, what about the use of brotli and zstd? Generally
much better, but not always. Each of these compression tools is configurable, and can be configured to be worse than
default gzip. Is each object, and in particular each static object, compressed with the maximum compression offered
by zstd or brotli? Who knows?!? Nobody checks... because nobody cares. In virtually all cases zstd is much faster
at compression, in addition to being much better at compression, that even the best gzip or best brotli. In all cases
it is faster to decompress -- which needs to be in order for humans to read the HTML. Is this site actually optimizing
and using the *right* zstd compression? Almost certainly not.

What about WOFF2? There are giant font packages here and they are delivered with WOFF2. What is going on?
Do we know if the big fat fonts delivered are actually used on the page, or are they completely useless? Unknown,
but the likelihood that the WOFF2 package contents are correct is low, if the use of gzip is to be used as a guage.
What about the compression in WOFF2? Sure, WOFF2 *can* be compressed, but compression is not required! Is
it here? If it is compressed, is it compressed with the gzip engine (as is permitted)? If so, why, and what compression
configuration is used? Perhaps it is compressed with the brotli engine? OK, if so, is it *properly* compressed with
brotli? Look up the brotli_params structure in the compression engine... because you cannot check after the
compression is done. The default compression is wrong for static WOFF2 objects.

All this is just check-marks in some GUI somewhere to define the contents in the delivered HTML stream. Is it
possible to just turn on modern compression here at caguns.net? All that I am talking about was news more than
5 years ago, but still sites like this are unable to get the basics right. The California 2A community wants modern
tools.

 

[ikeo1]

All that I am talking about was news more than
5 years ago, but still sites like this are unable to get the basics right. The California 2A community wants modern

We'll be updating to reduce some of the bloat in the near future. Welcome to CAguns. We leverage a lot of different technologies here to offset different limitations so your experience should be nothing like Calguns here.

Though the packages are a bit larger, you'll notice that we do cache things heavily so calls are minimized and page rendering is fast.

 

[-D.M.-]

 

[-D.M.-]

I keed, I keed...

 

[command_liner]

Cache is a good thing. But please do not cache poorly constructed objects. Construct the objects correctly to
begin with. Stop paying the price for transmitting too-big objects. Stop slowing down the user experience.

• Like Facebook and Google did, just give up on gzip. No more gzip permitted! That goes for advertisers too.
• Turn on the compression to the max with brotli and zstd. Serve compressed data. Zstd unless impossible.
• Do a dependency analysis on the fonts and deliver a WOFF2 object that contain ONLY the ones that will be used on the page.
• Configure your WOFF2 generator to use maximum brotli to minimize the overall size.
• Compress everything, not just some of the objects or data streams

Page rendering time does not, per se, depend on user-local or server-local cache. The javascript interpreter in your browser
is Turing-complete and overall execution time (time to render) depends on the Turing complexity. Sure, you can put
accidental bloat-ware in there, but there can also just be stupidly complex page layout doing dumb, slow things that
make rendering take a long time. Bloated or stupidly complex page design is almost always has a large byte count.
You want to minimize the byte count transmitted. That will also minimize the cache footprint on the user side. Also
think *hard* about complexity. (I know this is impossible to "fix" due to the halting problem, but you can do a lot to
get better. The US Patent Office even agrees with me!)

 

[ikeo1]

You want to minimize the byte count transmitted. That will also minimize the cache footprint on the user side. Also
think *hard* about complexity.

We actively do this. There's a lot of character and size limits every where on this platform. We also use a several optimizers including brotli and network acceleration and we also serve objects from edge CDN. I think you'll find there's not a lot of lag here and the site itself is very optimized for what we have.

From an administrative perspective, we've done what we can. From a software perspective, the next migration will reduce the payload you're talking about which is not within my control. We run as lean as we can. However, we also make sure we have extra cycles for just about everything, so again nothing should lag.

Page load with all objects is ~600ms with images, when cached it drops to 200ms but since we've added tracking pixel, it prolongs the full page load.

Once we perform the migration, this score will improve. So its on the radar but we don't need more optimization at this point. We're running pretty smoothly right now. When the system grows and after the migration, I'll run another set of optimization on the application services since system specs are increasing along with the user base.

How's your experience so far with the site? Are you noticing any lag at any parts of the system? Search results should return in < 10ms.

 

[command_liner]

Some things are OK, but nobody is paying attention to the easy stuff. For example, the biggest slowest
thing is a blob from cloudflare, vcd15cbe7772f49c399c6a5babf22c1241717689176015. It is a gzipped thing.
This exactly proves my point... get rid of that non-performant, excessively expensive crap.
Second slowest is a 5K image from your site. Each over 500 ms.

Just do a time Pareto and a space Pareto using Chrome. You can see the time-turds in ten seconds work.
Look to minimize the time-to-first-render, which is the key metric for human consumption.

If you are using cloudflare actively, your are doing it wrong. Turn off gzip and turn on zstd. It is right there
in the configuration GUI for your cloudflare access. If your class of service lets you configure this, turn off
the default compression configuration, and turn on more serious compression... for zstd, brotli, woff2 and
(if there is no choice) gzip.

Brotli (and zstd and gzip) are not optimizers but are just compressors. These are lossless and neither remove
nor add information.

 

[ikeo1]

Turn off gzip and turn on zstd. It is right there
in the configuration GUI for your cloudflare access. If your class of service lets you configure this, turn off
the default compression configuration, and turn on more serious compression... for zstd, brotli, woff2 and

I’m not familiar with what you’re describing. Can you take a screenshot? If you’re saying I’m doing it wrong without knowing my config, perhaps you can show the configuration params.

If a 5k image is slow it might be your computer. That shouldn’t take 500ms.

 

[command_liner]

Compression is important. Compression is money! Cloudflare knows that but has screwed up badly a few times.
But they are getting it right now.

They screwed up the implementation of zstd by using a python implementation that was bad. The complexity
of proper multi-threaded, multi-buffered compression is well known, and the work on zstd chosen by Cloudflare
was just not tested. Who knew that releasing untested multi-thread, multi-buffer compression would be bad?
Lots of us! When I wrote the first multi-threaded zlib implementation (like 30 years ago) it was really important
to get these details right... and that has not changed.

See https://blog.cloudflare.com/new-standards/#:~:text=Today, we are excited to,achieving nearly equivalent data compression.

All the way up to 11: Serve Brotli from origin and Introducing Compression Rules

Today, we're enhancing our support for Brotli compression, enabling end-to-end Brotli compression for web content. Compression plays a vital role in reducing bytes during transfers, ensuring quicker downloads and seamless browsing

blog.cloudflare.com

Compression Rules settings

Available compression algorithms and content type settings for Compression Rules.

developers.cloudflare.com

Compression Rules examples

Example Compression Rules for Brotli, Gzip, and Zstandard configurations.

developers.cloudflare.com

Enable Zstandard compression for default content types

Create a compression rule to turn on Zstandard compression for response content types where Cloudflare applies compression by default.

developers.cloudflare.com

A configuration that permits poorly compressed data to come over the wire is wrong. I do not need to see your config,
just the results of a page refresh.

Delays? Perhaps a pan-dimensional ziggle-bop is farting in your ear? That is about as likely as my computer being
slow. No, indeed one can just use chrome and look at the delay between when the object is resolved, and how long
the server (your server) waits before sending it. Sometimes it is a few ms, sometimes it is a few hundred.
Please check it yourself. It only takes ten seconds to do. You can run chrome in multiple virtual network speed
modes -- and those modes change the download speed. The resolution and wait for send to start is constant, and
excessive. Transmit time varies.

We are waiting 309 ms for https://caguns.net/data/assets/logo/caguns_white_200x75.png to *begin* transmit. After
taking 0.22 ms to resolve. Transmit takes 109 ms. Why all the waiting? This static object should have 0.1 ms wait
time. What are your servers doing? Going out for coffee? Or could it be that bloatware is getting in the way of
fast execution?

Almost an eternity -- 809 ms -- waiting for cloudflare to send gzipped css.php files. 16 ms to download and execute.

https://caguns.net/css.php?css=public%3Anormalize.css%2Cpublic%3Afa.css%2Cpublic%3Acore.less%2Cpublic%3Aapp.less&s=9&l=1&d=1731358916&k=6d1cf56fe6fc5bf3ffc93d8e920238bdf444e166

Fix that delay. Get rid of obsolete compression. Use zstd with custom dictionaries for compression of static objects.
If you have enough compute, do this for every visible and invisible piece of HTML that is streamed over the wire.
Think about it: Facebook created this feature for exactly this purpose. Were they wrong? If so, how and why? Be
specific: hand-waving generalities will not cut it in this discussion. Why wait a second for a static CSS object?

It is important to look at the components of delay for each of the top 20 objects -- by size and by time. In theory the
static zstd-compressed objects served by cloudflare should have essentially zero delay. What is going on? Is it
just poor quality of service from cloudflare or some other BS?

 

[ikeo1]

If you have enough compute, do this for every visible and invisible piece of HTML that is streamed over the wire.
Think about it: Facebook created this feature for exactly this purpose.

I've added this to my list to check out once we update. The css file will significantly reduce once we update along with a few other packages like jquery. For now, a lot of people don't notice the fraction of a second issue but it is an optimization point to review at a later time once we've upgraded.

We are planning to upgrade for other features mentioned in the migration plan so a lot of this is known but I'm sure there's some nuggets of info in your post.

For the time being performance is "good enough" and I have to move on to other projects and features.

Thanks for taking the time to do a performance audit.

 

[DB>)>]

Compared to caLguns, it's far superior to "gud enuf"... <wink>. I'm always happy when I see site designers that realize that not everyone has fast fiber, a T1 or T3 or whatever for "bandwidth/pipeline"!! Thanks again for providing a functional site, I'm sure there will be growing pains (especially since my observation on "users" on the "L" is that they are down about 75% just in the last few days...), but so far so good!

 

[DB>)>]

It's funny reading this thread, I'm reminded how much of a fossil I am - I gave up on coding a long time ago when I realized the "trend" was to what I refer to as "OTF programming" (alternately "On The Fly", or OH, That's F*****...) around 30 years ago. I saw the current game of "fixing the engines on an airplane midflight" (a LITERAL quote from a project manager of a major e-Com site!), and helped a few sites sort out their "oopsies"... my brain works well at isolating WHERE in the code the problems are, but I long ago stepped away from fixing the code! I leave that to my son...

Anyhoo, much appreciate that keeping a site up and running nowadays is a challenge, as evidenced by the slow motion trainwreck at "the L"... and I still hope they can eventually rise from the ashes as there's a lot of legacy info there among the remains!

And if it wasn't obvious my opinion on OTF programming... dinner tonight was a discussion with my other two kids about the issues with the Chrome browser, failed Windows updates, and the rediculous advert bloat on YT... along with ways to bypass the obstacles related to each! I pity those who just expect the magic box to work all the time without failure or error!

 

[command_liner]

Although I am pleased that caguns.net is working, I do wonder about the "gud enuf" programming mentality that permeates so much of the web today. In the example at hand, why would anybody make a website that has font references that *might* point to uncompressed, unneeded data? Especially when the font package is therefore the largest, slowest, least reliable, most expensive object on the page. It is the "gud enuf" mindset... and it applies to almost all websites. This one is no exception. People without 100 Mbit internet connections are just SOL.

I am working on a article "Complexity Breeds Confusion" about similar systematic failures in software systems today. There are plenty of big ones, and they have cost all of us stunning amounts of money.

 

[DB>)>]

I believe that part of it stems from a lot of code being cobbled together from bits and pieces, sometimes in a foreign language (like some stuff I saw posted in "the L" that was floating around their code base, and what I could see was definitely a problem!).

The new Chrome and Edge browsers for instance supposedly share the Chromium base, I've had lots of strange cross polination, weird glitching, and site/page load issues ever since those two "kissin' cousins" became the promoted flavors of browser...

While I'm sure that "modular" bits and bobs make for faster "simpler" development, it leaves open unexpected interactions, security issues, and inefficiencies. I've watched as it encourages that "OTF", fix it in the mix mindset which surprisingly works well... until it doesn't. I saw it coming around the time of Win95 development and testing, and it's become the common approach, great for fast adaptation to the market needs, but when it messes up, it can be pretty bad... almost makes one miss the days of vetted code, well tested Alpha and Beta phases before turning things loose in the wild...

All that said, CAguns has satisfied the (somewhat unexpected?) need for a stable, functional CA specific 2A site as "the L" fell apart, and there is plenty of time to "fine tune". Works pretty darn well for a "work in progress"!

 

[ikeo1]

I do wonder about the "gud enuf" programming mentality that permeates so much of the web today. In the example at hand, why would anybody make a website that has font references that *might* point to uncompressed, unneeded data? Especially when the font package is therefore the largest, slowest, least reliable, most expensive object on the page. It is the "gud enuf" mindset... and it applies to almost all websites. This one is no exception. People without 100 Mbit internet connections are just SOL.

You’re more than welcome to put your skills to the test like I did and create a site. I think you’ll find in the pursuit of technical perfection for which there never is, there are just as many important factors that aren’t technical and the ROI isn’t worth the time vs taking care of other more important issues to the user base.

I think you’ll find from a user perspective it’s always easier to find fault as opposed to being in the creator perspective when you try to juggle many different issues at the same time.

What I find is the issues you mention are over exaggerated for most users. 22ms here inconsistent 500ms there is not noticed with caching on, which is default. It’s old adage, if you’re looking for problems you’ll find them.

As mentioned several times, there’s an upgrade path while I continue to develop other areas. This site is not paying my bills. I have another business that makes more $$ that I’m building from scratch as well. While I enjoy working on this site, the amount of effort to get barely noticeable gain is quite large. It’s a time/value issue. Otherwise, you could/would build a site exactly the way you want.

I would also challenge you to investigate the cost of efficiency during growth of a stage. Should efficiency be the focus or growth because you have to slow one down to implement the other. If you’re saying you can do both, again, build a prototype with the features you see here and you’ll see just how complicated it gets. Then as an end user, I’m willing to bet I can find plenty of issues. Feel free to publish your paper here, I don’t mind reading it.

 

[bbp00]

What about WOFF2? There are giant font packages here and they are delivered with WOFF2. What is going on?
Do we know if the big fat fonts delivered are actually used on the page, or are they completely useless? Unknown,
but the likelihood that the WOFF2 package contents are correct is low, if the use of gzip is to be used as a guage.
What about the compression in WOFF2? Sure, WOFF2 *can* be compressed, but compression is not required! Is
it here? If it is compressed, is it compressed with the gzip engine (as is permitted)? If so, why, and what compression
configuration is used? Perhaps it is compressed with the brotli engine? OK, if so, is it *properly* compressed with
brotli? Look up the brotli_params structure in the compression engine... because you cannot check after the
compression is done. The default compression is wrong for static WOFF2 objects.

Origin of most fonts used in web applications is Google fonts. Google compresses all WOFF2 fonts with brotli. In rare cases, where origin (source) and compression differ, difference isn't large enough to bother, unless you're one of those mega sites, that few bytes can save you millions in bandwidth costs.

As for brotli vs zstd vs gzip, the answer is 'depends'. While they are benefits of one over the other, so are downsides. CPU cycles vs size, save few ms download time vs increase of TTFB, etc.

Image size, well, that's probably best to worry about when everything else is done. Easy enough to optimize, but you'll still stuck with images that users will post. It's a forum. It's a compromise.

All of the above I consider 'fine tuning'. Worthwile of doing if/when costs increases or user experience degrades.

As it is right now, it's fine. TTFB is low, and on my side, everything loads under 1 second. Currently, I don't think few ms will change user experience much if at all.

Remember, "the best is the enemy of the good".

 

[command_liner]

Any progress on this odd topic? I see that Cloudflare has a good summary of the trivial change required.
Check https://developers.cloudflare.com/rules/compression-rules/examples/enable-zstandard/
The stupid (and obvious) threading errors in python implementation appear to be fixed.

Once Cloudflare does the compression into zstd and then caches the object, there is just a constant
reduction in time. Time to transmit, and time to load, and time to uncompress in the user's browser.
Reduced cost and improved user experience ... all from proper configuration of the intermediate host.

The alternative is... use of a compression method that was obsolete decades ago. Slowed user
response. Higher costs.

Perhaps some customer here actually *disables* the use of zstd. It would be interesting to know the
number of requests per million HTML transactions that have zstd turned off. This statistic is available.
Is it as high as one in a million? One in a billion?