Ted Leung on the air
Ted Leung on the air: Open Source, Java, Python, and ...
Ted Leung on the air: Open Source, Java, Python, and ...
Tue, 06 Jan 2004
We need help with conditional GET
With the goal of helping blog software produce feeds that support conditional http GET and Gzip, I present a sampling of RSS feed URL's which fail to produce 304's. They also happen not to Gzip. In the table below, I show a semi-randomly selected feed URL and the blogging software which I believe is being used. I generated this list by hand, using NetNewsWire's bandwidth statistics window as a guide. Next time around I'll probably end up writing a script. But I hope there won't be a next time...
Jay at Makeoutcity http://www.makeoutcity.com/RSS/index.rss gets his own special category because his feed is *huge* (400kB a shot) and does neither 304 or Gzip, and I have no idea what blog software he is using.
If the JRoller/Roller and gotdotnet/weblogs.asp.net systems would support 304 and gzip, it would change my statistics dramatically, due to the number of feeds using those systems.
[01:31] |
[computers/internet/weblogs] |
# |
TB |
F |
G |
15 Comments |
http://crazybob.org/roller/rss/crazybob | Roller |
http://www.jroller.com/rss/ericfj | JRoller |
http://linuxintegrators.com/hl40/blog/?flavor=rss | Blojsom |
http://blogs.gotdotnet.com/aconrad/BlogxBrowsing.asmx/GetRss? | Gotdotnet blogs - BlogX |
http://weblogs.asp.net/MainFeed.aspx | Weblogs.asp.net - .Text |
http://www.25hoursaday.com/weblog/SyndicationService.asmx/GetRss | dasBlog |
http://www.cincomsmalltalk.com/rssBlog/rssBlogView.xml | Whatever the Cincom blogs use |
http://www.vanderburg.org/cgi-bin/glv/blosxom?flav=rss | blosxom |
Check out the Cacheability Engine; it checks for conditional support, caching headers, clock skew, etc.
I'm about to do a rewrite to squash some long-standing bugs and generally make the code more sane, and I'll see if I can get gzip support in as well (so if you have another other requests/comments, now would be a good time ;)
Posted by Mark Nottingham at Tue Jan 6 07:47:21 2004
I'm about to do a rewrite to squash some long-standing bugs and generally make the code more sane, and I'll see if I can get gzip support in as well (so if you have another other requests/comments, now would be a good time ;)
Posted by Mark Nottingham at Tue Jan 6 07:47:21 2004
Hrm, on makeoutcity.com it's apache with mod_gzip, or so says lynx -head. It is returning ETag correctly, so news readers with ETag support should not download the beast again.
Posted by Patrick Berry at Tue Jan 6 08:50:37 2004
Posted by Patrick Berry at Tue Jan 6 08:50:37 2004
It almost goes without saying, but I'll say it anyways, that anyone using Moveable Type (or any other software which publishes statically) defers these capabilities to Apache (or their webserver of choice), so anyone using MT should at least support conditional get. (mod_gzip is kind of hairy, and obscure)
Posted by kellan at Tue Jan 6 11:26:18 2004
Posted by kellan at Tue Jan 6 11:26:18 2004
Ted, blojsom now supports conditional GET. Previously we only had the GZIP filter in place. The conditional GET plugin was just added a few days ago, but I've got it running on my blog and it works like a champ. The GZIP filter has been available since early November and is enabled by default with the blojsom WAR I package up with the distribution.
<p/>
Whether or not choose people upgrade to the latest version of the software and use the latest plugins when available is something I can't control. I can only put things out there in hopes that people "do the right thing".
Posted by David Czarnecki at Tue Jan 6 15:23:17 2004
<p/>
Whether or not choose people upgrade to the latest version of the software and use the latest plugins when available is something I can't control. I can only put things out there in hopes that people "do the right thing".
Posted by David Czarnecki at Tue Jan 6 15:23:17 2004
Roller has supported gzip (with a Filter) and 304 for some time. I don't know which version of Roller CrazyBob is using, but I think think JRoller has deferred gzip to Apache. There were some issues with conditional Gets at some point in the past, but I don't recall at what point Dave Johnson fixed them.
Try http://www.brainopolis.com/roller/page/lance which is running a fairly current version of Roller.
Posted by Lance at Tue Jan 6 20:41:35 2004
Try http://www.brainopolis.com/roller/page/lance which is running a fairly current version of Roller.
Posted by Lance at Tue Jan 6 20:41:35 2004
Livejournal is another one - the RSS feeds aren't gzipped, but the main diary pages are, so for a simple "update" check it's better to fetch the gzipped main page than the RSS. Also, it doesn't seem to support 304 or Etag in my brief (but possibly inaccurate using lynx) testing.
Posted by TRS-80 at Tue Jan 6 22:32:14 2004
Posted by TRS-80 at Tue Jan 6 22:32:14 2004
Mark,
Here's a site with a perl script for testing gzip
http://www.neuro-tech.net/archives/000031.html
Cacheability Engine looks really good!
Posted by Ted Leung at Tue Jan 6 22:37:28 2004
Here's a site with a perl script for testing gzip
http://www.neuro-tech.net/archives/000031.html
Cacheability Engine looks really good!
Posted by Ted Leung at Tue Jan 6 22:37:28 2004
Patrick,
My bad on Etags for Makeoutcity. It wasn't gzipped, though. Having mod_gzip != using mod_gzip. It should be fixed now. I talked to Jay about it in #joiito today.
Posted by Ted Leung at Tue Jan 6 22:38:25 2004
My bad on Etags for Makeoutcity. It wasn't gzipped, though. Having mod_gzip != using mod_gzip. It should be fixed now. I talked to Jay about it in #joiito today.
Posted by Ted Leung at Tue Jan 6 22:38:25 2004
David,
I know that you don't have control over plugins. But Andy should know better. Perhaps you could put a big notice in the installation docs, and the announcement for 2.08 so that people become aware of the issue.
Posted by Ted Leung at Tue Jan 6 22:40:24 2004
I know that you don't have control over plugins. But Andy should know better. Perhaps you could put a big notice in the installation docs, and the announcement for 2.08 so that people become aware of the issue.
Posted by Ted Leung at Tue Jan 6 22:40:24 2004
Lance,
If you put your RSS url into Mark's Cacheability engine http://www.mnot.net/cacheability/, you'll find some interesting results.
There've been a number of JRoller related problems for me over the months. I still keep getting a full copy of Carlos V.'s feed every day. Since this isn't happening with many of the other feeds, that leaves JRoller as the culprit.
Posted by Ted Leung at Tue Jan 6 22:50:17 2004
If you put your RSS url into Mark's Cacheability engine http://www.mnot.net/cacheability/, you'll find some interesting results.
There've been a number of JRoller related problems for me over the months. I still keep getting a full copy of Carlos V.'s feed every day. Since this isn't happening with many of the other feeds, that leaves JRoller as the culprit.
Posted by Ted Leung at Tue Jan 6 22:50:17 2004
Thanks Ted - good post. I've been meaning to drop the 304 stuff into Pebble for a while but never got around to it. It's working locally but I'll have to wait until later when I can upload a new WAR file to my ISP.
Posted by Simon Brown at Wed Jan 7 07:06:15 2004
Posted by Simon Brown at Wed Jan 7 07:06:15 2004
Ted, I'd suggest filing a bug report about Roller; it seems that it should support it, yet you've identified that it isn't. Perhaps someone on the development team would have more information -- and getting the issue on their rader in any case seems to be a good plan.
Posted by Richard Soderberg at Wed Jan 7 11:55:36 2004
Posted by Richard Soderberg at Wed Jan 7 11:55:36 2004
There has been some discussion of late about RSS readers causing the downfall of mankind. This has been discussed before on many occasions. Being that I like participating in the great world wide web as a responsible netizen I activated gzip encoding of my feeds and other content many moons...
Posted by Trackback from randomthoughts at Tue May 4 10:35:06 2004
Posted by Trackback from randomthoughts at Tue May 4 10:35:06 2004
There has been some discussion of late about RSS readers causing the downfall of mankind. This has been discussed before on many occasions. Being that I like participating in the great world wide web as a responsible netizen I activated gzip encoding of my feeds and other content many moons...
Posted by Trackback from randomthoughts at Tue May 4 10:39:57 2004
Posted by Trackback from randomthoughts at Tue May 4 10:39:57 2004
You can subscribe to an RSS feed of the comments for this blog:
Add a comment here:
You can use some HTML tags in the comment text:
To insert a URI, just type it -- no need to write an anchor tag.
Allowable html tags are:
You can also use some Wiki style:
URI => [uri title]
<em> => _emphasized text_
<b> => *bold text*
Ordered list => consecutive lines starting spaces and an asterisk
To insert a URI, just type it -- no need to write an anchor tag.
Allowable html tags are:
<a href>
, <em>
, <i>
, <b>
, <blockquote>
, <br/>
, <p>
, <code>
, <pre>
, <cite>
, <sub>
and <sup>
.You can also use some Wiki style:
URI => [uri title]
<em> => _emphasized text_
<b> => *bold text*
Ordered list => consecutive lines starting spaces and an asterisk