LUI: episode III – The ISP Empires have Technical Advantages
Following on from Jeremy’s second episode of our journey into the life of an imaginary small ISP, I thought I’d explain how technically challenged we are compared to the larger brethren.
The major challenge with software development is that incremental costs of delivery are very slight and having a large user base is actually very good for rapid bug finding and diagnostics. In an ISP this results in favouring those ISPs with a large customer base who can spread development costs over a large number of customers. This assumes that ISPs are equally proficient at software development which is not always the case: there are some very good case studies from the ISP community on how not to develop software and the distribution of these disaster projects seem to be proportional to ISP size. However, even with occasional cock-ups the dice are firmly loaded in the favour of the larger ISPs.
For this reason, LUI has always taken the approach using third parties wherever possible for software services. We bring to the party our customers and our trusted relationship with them. We tend to focus on integrating our customer database and billing software with the third party software, so provisioning and billing are transparent to the end user.
Our software development team is only six strong and that includes a manager who we force to get his hands dirty with periodic bug squashing as well as playing with project plans on Excel and practicing motivational speeches in front of the mirror. One of the golden rules of our product lifecycle is that apps are only launched when self-provisioning is possible and integrated with our portals and all the tools for app monitoring are available online for remote use by our NOC.
LUI also has an allergy for paying for software and prefer to engage in revenue sharing deals when services are sold to our base, open source apps when we need to host our own services and as a last resort we roll our own. LUI buys plenty of hardware, but it is normally off the shelf last quarter’s model rather than the latest, greatest and most expensive bits of kit. LUI even stretch this philosophy to our routers and switches where we have no alternative but to buy proprietary kit. Even with this approach, because of our lack of purchasing power compared to the larger ISPs we know LUI still pays a premium for hardware.
Our Sysadm team is a mere seven strong and includes specialist support for our radius/ldap, proxy, provisioning, billing and traffic management apps. We also have to deal with our third party interfaces and have a specialist in core network traffic planning. The obligatory manager comes along for the ride and who strangely seems to spend most of his time with our third parties doing lunch and discussing theoretical issues.
Our lack of scale also works against us in Operations: our NOC team is seven strong and that includes a manager and a scheduler. Somehow we manage to eke out 24/7 coverage. We have to train multi-skilled engineers who not only can sit and monitor machine performance and occasionally fix problems, but also they need to double up as field engineers who can visit PoPs and upgrade equipment.
A larger ISP doesn’t have these problems and can easily afford specialists in their NOC covering UK wide networks and even have the scale to economically outsource field engineering.
LUI’s twenty strong technical team may not sound like a lot, but costs us plenty: the average salary of the managers is £50k/annum and the other seventeen averages out at £30k/annum. This is £660k per annum just for team, before a training budget of £100k and team equipment and consumables of another £100k. In addition, we tend to spend just on IT hardware for our server farm of around £300k per annum and our server farm incurs maintenance and colo costs of around £40k per annum.
The opex element of these costs (£900k) come straight off our bottom line and work out costing us far more than our backhaul costs on the unbundled side of the operation. When we spread the costs equally over our 47k unbundled and around 53k ipstream customer – it works out at 75p/customer/month. With the capex costs being more or less continuous every year we can add another 25p/customer/month for these.
Basically the only way of reducing our unit technical costs is to gain scale quickly.
The major challenge with software development is that incremental costs of delivery are very slight and having a large user base is actually very good for rapid bug finding and diagnostics. In an ISP this results in favouring those ISPs with a large customer base who can spread development costs over a large number of customers. This assumes that ISPs are equally proficient at software development which is not always the case: there are some very good case studies from the ISP community on how not to develop software and the distribution of these disaster projects seem to be proportional to ISP size. However, even with occasional cock-ups the dice are firmly loaded in the favour of the larger ISPs.
For this reason, LUI has always taken the approach using third parties wherever possible for software services. We bring to the party our customers and our trusted relationship with them. We tend to focus on integrating our customer database and billing software with the third party software, so provisioning and billing are transparent to the end user.
Our software development team is only six strong and that includes a manager who we force to get his hands dirty with periodic bug squashing as well as playing with project plans on Excel and practicing motivational speeches in front of the mirror. One of the golden rules of our product lifecycle is that apps are only launched when self-provisioning is possible and integrated with our portals and all the tools for app monitoring are available online for remote use by our NOC.
LUI also has an allergy for paying for software and prefer to engage in revenue sharing deals when services are sold to our base, open source apps when we need to host our own services and as a last resort we roll our own. LUI buys plenty of hardware, but it is normally off the shelf last quarter’s model rather than the latest, greatest and most expensive bits of kit. LUI even stretch this philosophy to our routers and switches where we have no alternative but to buy proprietary kit. Even with this approach, because of our lack of purchasing power compared to the larger ISPs we know LUI still pays a premium for hardware.
Our Sysadm team is a mere seven strong and includes specialist support for our radius/ldap, proxy, provisioning, billing and traffic management apps. We also have to deal with our third party interfaces and have a specialist in core network traffic planning. The obligatory manager comes along for the ride and who strangely seems to spend most of his time with our third parties doing lunch and discussing theoretical issues.
Our lack of scale also works against us in Operations: our NOC team is seven strong and that includes a manager and a scheduler. Somehow we manage to eke out 24/7 coverage. We have to train multi-skilled engineers who not only can sit and monitor machine performance and occasionally fix problems, but also they need to double up as field engineers who can visit PoPs and upgrade equipment.
A larger ISP doesn’t have these problems and can easily afford specialists in their NOC covering UK wide networks and even have the scale to economically outsource field engineering.
LUI’s twenty strong technical team may not sound like a lot, but costs us plenty: the average salary of the managers is £50k/annum and the other seventeen averages out at £30k/annum. This is £660k per annum just for team, before a training budget of £100k and team equipment and consumables of another £100k. In addition, we tend to spend just on IT hardware for our server farm of around £300k per annum and our server farm incurs maintenance and colo costs of around £40k per annum.
The opex element of these costs (£900k) come straight off our bottom line and work out costing us far more than our backhaul costs on the unbundled side of the operation. When we spread the costs equally over our 47k unbundled and around 53k ipstream customer – it works out at 75p/customer/month. With the capex costs being more or less continuous every year we can add another 25p/customer/month for these.
Basically the only way of reducing our unit technical costs is to gain scale quickly.
<< Home