2007-vftgs

Integrating the website

posted Jul 24, 2011, 5:22 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:23 PM ]

Wednesday, December 12, 2007, 10:16 AM
Posted by Administrator
A ship in port is safe; but that is not what ships are built for. Sail out to sea and do new things.
- Grace Hopper


My website worked and it was not completely unattractive. It had tools, some content and hopefully a little humor. Yet, a website should not be static, the same thing forever, it should grow and become more useful, more interesting over time. So, my website has changed and I'm about to make the pages I've been working on become the primary pages.

The changes are mostly the rebuilding of three wikis as a single wiki. This enables almost all the content to be searched from one location and simplifies management. I've learned a bit about how to use the tools in the wiki over time, and this gives me a chance to do better what I'd done before.

It also gave me a chance to dig into the guts of the code a bit and learn how it really works. The program is not so complex that it is impossible to fully comprehend but it defies minor tinkering and so there was a good bit of reading involved as well. I was able to do what I aimed for though, that being to integrate my own PHP code into the system, building it dynamically within the confines of the CMS. Essentially I can now call any custom designed (and pre-approved) function of my own creation with <?php somefunction?> tags inserted into the wiki content.

Regular readers will also notice the Blog has been reformatted to be closer in color theme and layout to the primary pages. It is a completely different piece of software and in some ways as complex as the wiki. It could not be simply tacked into the wiki and had to be manually adjusted on several levels to get it to both appear to be a part of the same site and also contain the navigation configuration I like. As much as I like both, they do not play nice. I'd like to consider rebuilding significant portions of the blog and integrating them into the existing blog code and then tie that into PMWiki, but the work involved to take such disparate coding systems and rebuild so much of each is somewhat daunting. Perhaps another day.

I've also removed all the javascript that made the blog fade in and out. Essentially I decided it was too little payoff for too much bandwidth. It is a cool idea, but probably best not to use it that way.

Rings Rings Rings

posted Jul 24, 2011, 5:21 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:22 PM ]

Monday, December 10, 2007, 07:58 AM
Three Rings for the Elven-kings under the sky,
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.
- J R R Tolkien


Around a year ago, I realized that hand coding my website was taking more time than it would to use a good CMS. I decided to replace the habits I had of using vim to build all my web pages and put the content into a different system. I wasn't happy about the overhead of the main wiki (MediaWiki) I'd been using, and found PMWiki to be a better option. MediaWiki is powerful and has a rich feature set and is widely adopted, but it relies on a database system, which I didn't really want. I know from past experience that a personal site with meager funding can't afford to be too picky about hosting and thus there is a chance that if I lock myself into a database system, one day it might not be there at all. PMWiki doesn't use a database system, instead it uses files to manage content and indexing. Its widely used (although not as widely as MediaWiki) and the syntax isn't too tough to learn. Overall, it has served me well.

The next step was to gather content and put it into the wiki, but there was a pretty ungainly bundle of web pages on the site, and a lot of them used common resources and customized scripts I'd written over the years. To deal with that, I started gathering them into three groups, computer stuff, general writing and entertainment. In the process, each got its own wiki system and eventually pretty much everything that could be bundled had been bundled.

Now I'm consolidating the three wikis into a single wiki. This should make the entire site much easier to manage and make it search friendly. It will take some time, but I'm well on my way now. The trickiest bit is maintaining old paths that have been linked to externally. I have previously been hand coding wiki-like pages to fill the locations, but I'm going to switch to redirects shortly.

So I'm taking my three wikis, which were essentially binding together the site and combining them under one new one. Tolkien would be flattered I think.

I am a computer nerd

posted Jul 24, 2011, 5:20 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:20 PM ]

Thursday, December 6, 2007, 12:38 AM
Posted by Administrator
I think a nerd is a person who uses the telephone to talk to other people about telephones. And a computer nerd therefore is somebody who uses a computer in order to use a computer. - Douglas Adams

Lets say you're staying at my neighbor's house and you fire up your laptop and jump on the wireless. Oops, you got my wireless, and why not? It's completely insecure after all. You check your gmail and check yahoo news, but then you try to go to myspace.com and you get some other page instead! Yes, my temporary neighbor, you're being blocked. My wireless now allows anybody on, and allows surfing to a list of pre-approved sites, but only those sites until you perform some other authentication. If you should happen to try to access something not white listed, you get a friendly replacement page with a handy list of pre-approved sites.

Its magic! Actually, it's a combination of OpenVPN with squid proxy and some pretty complex IP Tables firewall settings. Essentially it works this way, the wireless router will let anybody on, but the firewall machine assigns the DHCP with itself as the default route. Of course, you might be able to route yourself through the wireless router, but it makes no difference. Once you are connected, all traffic on port 80 is redirected through a transparent squid proxy, which allows traffic monitoring and more particularly allows only browsing to a specific list of domains. If you try to access something not in that list, you get the squid error message, the custom one that frames a page presented by my second http server. I'm running two because there is significant work on the main one, and only minimal interaction on the second so the first is Apache and the second is thttpd (tiny httpd.)

Periodically a cron job runs to update the list of sites in a file available to the thttpd server. It presents an error when you hit anything that doesn't exist, and the only other pages are one that builds the list (cgi written in C) and the index, which frames the error.

Its all invisible to the end user of course, as far as they see, they can only browse my approved sites through my wireless network, as a guest might expect, and they don't have to know anything at all to use it.

Next on the agenda is to set up a mailing system so that guests can request openvpn encryption keys or addition of whitelisted sites. I of course, already have an encrypted tunnel and therefore I can do pretty much anything I feel like on it.


"What really is the point of trying to teach anything to anybody?"
This question seemed to provoke a murmur of sympathetic approval from up and down the table.
Richard continued, "What I mean is that if you really want to understand something, the best way is to try and explain it to someone else. That forces you to sort it out in your mind. And the more slow and dim-witted your pupil, the more you have to break things down into more and more simple ideas. And that's really the essence of programming. By the time you've sorted out a complicated idea into little steps that even a stupid machine can deal with, you've learned something about it yourself. - Douglas Adams

IP Tables, Firewalls, OpenVPN and the DI-524

posted Jul 24, 2011, 5:18 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:19 PM ]

Tuesday, December 4, 2007, 11:41 AM
Posted by Administrator
Know ye therefore that they which are of faith, the same are the children of Abraham. And the scripture, foreseeing that God would justify the heathen through faith, preached before the gospel unto Abraham, saying, In these shall all nations be blessed. (Gal 3:7-8)

As a justified heathen, I say I am blessed to be one.



I have a network at home, about eight or so computers, but really three regular use ones and I don't care for stringing wire, and I have a wireless router, so making it work is important to me. Now it is possible, without too much trouble, to use an encrypted session (using WPI PKI not WEP, WEP is easily hacked, WPI is much safer) but that means that I have to have a long key handy whenever I want to log in, and hope that my particular card and OS support it. Typically Linux does support it, but I do try out various distributions from time to time and getting encryption working between different hardware can be tricky. There is a better alternative: introducing OpenVPN! (and you should hear a crowd cheering in your imagination at this point.)

OpenVPN is a way to tunnel all your traffic through a network connection, wireless or wired, and it works on Windows, Mac and most Linux, plus it is software so you don't have to do any special hardware work beyond the minimum. Essentially, it gives you a method of keeping your traffic secure without having to worry about special drivers.

I'm using it to let my wireless network be totally open (well, the admin of the router isn't but that's about it.) This means that I can connect to my wireless network without needing any passwords or encryption at all. It also means that anybody driving down my street or visiting the neighbors could too. I'm a friendly guy, and I do try to be helpful, but I'd rather not worry too terribly about what my Internet connection is being used for, so I'd rather that I was the only one allowed to use it. This is where a good firewall comes in. If the connection from the wireless network has to come in on VPN because the firewall blocks anything else, that means that VPN is now required.

Enough with the theory, the application was not bug-free. In fact, after much testing, I had found that I had no trouble using OpenVPN between machines on the wired network, but a whole heap of trouble when I tried to use it on the Wireless. Eventually, with a lot of testing and tracking packets, I was able to confirm that it seemed packets could come from the wireless, talk to the VPN, but then they couldn't come back across the wireless to the machine making the requests. It might be possible, but it is certainly not easy to change that behavior on the DI-524 D-Link wireless router I have.

So eventually, rather than trust the thing to manage my network, I just bypassed it by putting the OpenVPN (and Internet Gateway) server on the "LAN" side of the router. This means that now people connecting to the Wireless router can communicate directly with the OpenVPN server, without going through the standard router manipulation. It also meant I had to disable DHCP on the Wireless router since it couldn't make anything but itself the gateway and enable it on the interface that connected the OpenVPN server to the LAN side of the router. Not satisfied with that though, I set them both to be DHCP servers since the OpenVPN server seems to get precedence and just set them to assign non-conflicting ranges. Now it is possible to get on the Wireless even if the OpenVPN server isn't doing DHCP correctly, so it makes it possible to troubleshoot, but typically the OpenVPN server is doing fine.

Once I passed those hurdles, I have no trouble getting a wireless connection, and it does DHCP assignment as you'd hope, DNS assignment (and forwarding) and all that good stuff, but I still want to lock out the neighbors. Actually, I'm trying to lock out anybody that might be using it for anything illegal, but how well do you know your neighbors really, do you know their friends? So it is time for some healthy paranoia.

In with IP Tables. Here, I'm not really done. I've got something that works, but not as perfectly as I like so I'm going to refrain from passing on any advice on exactly how your firewall should look, suffice it to say that there are plenty of tools and manuals out there for you to consult, but I do have some other advice:

Suggestion 1:
Freenx and Nomachine rock, try them. It's like VNC (remote computer management) but smoother and better. Nomachine is clear that their product is not the same as Freenx, but you can use them together so I do. I would recommend trying Nomachine first if you have the option, since I had some initial trouble getting documentation. I had to do some experimenting to find out exactly how to set everything up with FreeNX, but if you're comfortable with that sort of thing, it isn't too bad. Of course you don't really need a GUI interface for your server, but why not have one if it is easy and, as far as I've been able to tell, secure. Freenx and Nomachine have a good system set up, using public/private key based ssh tunneling.

Suggestion 2:
Set up a simple "fix my router script" when working on your network remotely. For me, I have a script I call "insecurerouter.bash" which pretty much opens the system up to all traffic. I keep that machine pretty safe anyway, so it shouldn't be easy to compromise, but it does mean that anybody can connect to it or the Internet through it. I then set up an infinite looping script to check for the presence of a file every five minutes, and reset the router if it is there. Then I run another script on the connecting machine to remove it up to once a minute. One the router it looks like:
#!/bin/bash

while true
do
touch /home/unprivuser/.routerreset
chown unprivuser:unprivuser /home/unprivuser/.routerreset
sleep 300
if [ -f /home/unprivuser/.routerreset ]
then
echo "RESET ROUTER AT `date`"|wall
/root/bin/insecurerouter.bash
else
echo "No reset at `date`"
fi
done
and on the other machine I've got password-less ssh login set up (using encrypted keys and keyring to keep them in memory) and it has a screen session running with the command:
(while true;do date;ssh n 'rm -v ~/.routerreset';sleep 60;done) &
also inside a screen session. Sure, there is a little verbosity there that isn't really required, but sometimes it is nice to be able to check on what's happening, and screen has a very low overhead. Essentially, this means then when I'm testing a new firewall, if I manage to lock myself out, no more than five minutes later, I can log back in and try to figure out what went wrong. If you are building your own firewall, things will go wrong, and that leads to my next suggestion.

Suggestion 3:
This one is about IP Tables, and while I don't feel qualified to preach on the proper usage (I'm not currently filtering OUTPUT) I do feel it is worth noting that it is MUCH easier to troubleshoot if you log every unidentified packet you drop. That means that you can check your logs when something doesn't work and see if it is due to your rules, and if you're hand-coding your rules like I am, that is a lifesaver.

Blog modified

posted Jul 24, 2011, 5:17 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:17 PM ]

Monday, December 3, 2007, 01:01 PM
Posted by Administrator
This entry was previously part of the site news section and was moved to the blog later.

My site has included a blog for a while now, but I've not been happy with the way it was integrated with the main site. I've now got it scripted to fade and disappear with a Hide link, so it should be friendlier. I've also had to put in verification of comments due to a surprising level of spam.

Another change to the website

posted Jul 24, 2011, 5:16 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:17 PM ]

Friday, November 30, 2007, 08:56 PM
Posted by Administrator
I changed the site to use a javascript managed window slide but it relied on frame resizing, and I couldn't stand that. Now it still uses a frame, but it's an inline frame and the hiding is managed by javascript controls on the div tags manipulating the opacity. The biggest trick is trying to guess how it will work in other browsers, so YMMV. I did try to deal with that by handling both opacity and filter settings, and also putting in modifiers to change both the delay between increments of change and degrees of change so hopefully it will work on all browsers ever created. And, if it doesn't, well my site doesn't require javascript, and non-javascript enabled browsers can just use the in page standard html links since the link to use the javascript fading controls is itself reliant on javascript to display.

I can't decide whether to hate or love javascript.

Revisiting the GUI origins

posted Jul 24, 2011, 5:15 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:16 PM ]

Monday, November 12, 2007, 10:07 PM
Posted by Administrator
I should be asleep right now but I was just revisiting my article on the origins of the GUI and I thought it worth a moment to restate a quick summary here.

ARPA - Had a GUI and Douglass Englebart (sp?)
Xerox PARC - Got Douglass and made a GUI and personal computer at $40,000.
Apple - Had Steve Jobs who had stock in Xerox and visited and later released their own GUI
Microsoft - Made deals with Apple and then came out with a competing product

Lawsuits all around.

Now you can buy a home computer with Apple, Microsoft or Linux and all have an easy to use GUI.

FYIs:
The other major player in the OS world is Unix, which came out of Bell labs, but is now owned in part by several players.

Linux is not Unix, but it looks and feels a lot like Unix
Mac OSX Leopard is actually Unix and it feels a lot like Linux
Windows is the only OS that doesn't have strong ties to Unix and the argument could be made that the heavy development lifting done for Windows XP was done by Unix developers

I bring this up because I am typing this from an Apple and will probably be working on a Unix machine tonight. I'm going to head off to dreamland for a couple hours and I imagine that I'll be thinking somewhat of the question what makes Unix so special?

I suspect it is in large part because it is source code that is so widely spread it is nearly open. Linux is certainly due at least in part to the freedom that comes with Open Source software and BSD is nearly Unix, and in fact the Apple system is based on a BSD version that has been certified to now be Unix. AIX is what runs our major systems, and it is Unix.

What else is out there? There are a couple interesting players:
ReactOS - Open source Windows XP compatible variant, not quite ready for prime time.


Anyway thats enough for tonight.

Walmart ushers in the year of the Linux desktop

posted Jul 24, 2011, 5:14 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:15 PM ]

Monday, November 12, 2007, 09:42 PM
Posted by Administrator
Go Walmart, Go Walmart, Go Walmart, It's your birthday, It's your birthday
They're sold out of Linux computers!

Okay, a lot of people think Walmart is evil, but I'm far from convinced. I am convinced that they are run by savy marketing people with a keen sense of how to turn a profit.

If I could guess at their overall strategy though it would be:
Buy cheaper, transport cheaper, sell cheaper and sell a whole frickin lot!

Now I just read the news that they sold out of the $200 Linux Desktop machines they're selling.

I knew they were selling them but it sounds like they didn't even get to the shelves before they were ordered out of stock online.

It's the year of Linux people! I didn't think it would happen this year, but if Walmart started planning to stock and then sold out of them, it is the year of the Linux desktop.

Bugfree code

posted Jul 24, 2011, 5:13 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:14 PM ]

Monday, November 12, 2007, 07:41 PM
Posted by Administrator
Okay, so now my site is up for public review. It did not however get here as smoothly as I expected. First, I noticed that there were a lot of files not being uploaded and I've been using age controls so it turns out that many files have not been modified in a long time. I fixed that with find ./ -exec touch '{}' ';' and of course, now it had to upload every thing. That took me to the next step of setting an undocumented argument so that it could skip to the number of the file that it left off on. Finally, I finally had a timeout problem, and sure enough, it was not working correctly since I was not resetting the disconnected flag after reconnecting, essentially making a new connection for each file after a disconnect.

Now of course, it's bug free... this time. For those of you who don't recognize my phrase, it's what I now commonly use to indicate that while I've done my best to write a good program, I haven't tested it well enough to find out where the problems are, and there are always problems.

Now the page is busy

posted Jul 24, 2011, 5:12 PM by Boyce Crownover   [ updated Jul 24, 2011, 5:13 PM ]

Monday, November 12, 2007, 05:20 AM
Posted by Administrator
I've now added some scripting to the site and I think I can be happy with it. When you hit the index page now, the blog and normal pages are loaded in frames. If you have javascript enabled then the frames will be resized so that the blog is the full screen with the regular site hidden and then the regular page will seem to scroll over on top of the blog very quickly. I added a regular link to the blog on the main page and a javascript revealed link that scrolls the main pages back out of the way to the main menu bar. Essentially, the page remains as functional as it was, but now it has some javascript to make it obvious there is more content to see if you click the something. I'd like to ake it blink while the bar is scrolling away, but I don't feel it is worth the effort at the moment. I had to tweak the temlate I'm using a little though, now the "Hide Blog" link points to the main site. It's not quite as snappy as javascript all around, but I use noscript so I imagine others do too.

1-10 of 14