There's a sense of pride and satisfaction knowing how to run your own infrastructure, to splice your own fiber lines, and have your own website fly.

Recently, I've been working on redesigning and rebranding my own infrastructure, it was a mixture of vanity names that are inconsistent, and a lot of off-site systems. My main server is a 64 G / 4 TB VMWare Hypervisor, which does the trick, but in order to speed up connections to China (where ~35% of traffic originates), the line to my east coast server is terrible. Here we meet Burnaby BC, a place where the connection to China is about 120ms, versus to east coast, which is about 440ms.

My infrastructure (including this blog you're reading!) is powered by two reverse proxies, one in Burnaby, and one in Quebec. For most of Europe, you'll hit my Quebec proxy, but for the rest, you'll land on the other side of Canada in Burnaby. Burnaby has great connection to my main server (~40ms), so what I've started doing is aggressive caching. The content delivered over there is 100% static, I have no intend to change very often, so this blog for example, albeit running Ghost and being dynamic will only change maybe a few times a month. Using my edge proxy, I've setup a tiny script to curl my website once every half hour, and replace the contents of the Burnaby node's cache - this now serves the China visitors well.

OK, but what about dynamic data you can't cache?

In this case, it becomes more difficult - I start sending the Burnaby node the static files I can cache (CSS/JS/Images), and my CMS I wrote in PHP with Laravel framework will detect it's origin as Burnaby, and not write any CSS/JS/IMG to the view, this allows me on the edge in Burnaby to inject local copies of it, the URL path is rewritten from: static.domain.tld to static.edge02-burnaby.domain.tld - as you can see, it forces it into the Burnaby copies, keeping it from doing more round trips then it has to.

It's not perfect, but it works. Of course, I could fork over the money for a CDN, but my sites are not very intensive enough for the need, I can get by with just two servers on the opposite sides of the country. Peering with internet exchanges at both ends helps speed up the traffic as well, I'm always open to peering, so if you would like to peer send me an email, and let's peer. For those who cannot peer over regular BGP, you can peer with me over virtual networks (ZeroTier/OpenVPN/etc), and I can send routes your way.

engineering cloud bgp

Mike

Senior Software Engineer, Labber, Sysadmin. I make things scale rapidly. Optimize everything.

Read More