BHUTAN

MIGRATION TO

NEW TECHNOLOGIES

[WIRELESS VOIP]

MISSION REPORT

By

Clif Cox

Clif@eugeneweb.com


For Bhutan Ministry of Communications




Prepared by

International Telecommunication Union





October 30, 2002

$Date: 2003/12/03 00:09:57 $
$Revision: 1.0 $


Raising a Repeater in Rural Bhutan
Photo by Tensin Tobgyl

Abstract

A pilot project to use wireless and VoIP technologies to deliver communication services to rural areas in Bhutan, a small Himalayan Kingdom, was completed with encouraging results. Once initial problems with radio interference from other sources were solved the 802.11b radio network became reliable. This allowed the  VoIP equipment to be tuned to accommodate the more variable nature of a wireless network as compared to a wired one. International calls through the PSTN were hampered by a slightly non standard R2 protocol spoken by the local switch. This underscores the importance of adhering to open standards when many subsystems must work together.

Table of Contents

1 Background
1.1 Voice Communications
1.2 Data Communications
1.3 Combining Wireless and VoIP
2 Goals
2.1 Objectives
2.2 Expected outputs
3 Preliminary work
4 Project Sites
4.1 General guidelines for site layout
4.2 Backbone
4.3 Last mile
4.4 Network layout
5 Equipment choices and constraints
5.1 Radio
5.1.1 APs/Bridges
5.1.2 Wireless Ethernet converter
5.1.3 Amplifiers
5.1.4 Antennas
5.2 Mounting hardware
5.3 Supplies
5.4 Linux Routers
5.5 VoIP
5.5.1 Billing
5.6 Power
5.6.1 Commercial
5.6.2 Solar
5.6.2.1 Sizing of solar systems
5.6.2.2 Batteries
5.7 Timers
5.8 Weather proof enclosures
6 Grounding and Lightning Protection
6.1 Further recommendations for lightning protection
7 Shipping Considerations
8 Assembly
8.1 Timers
8.2 Repeater boxes
9 Site preparation
9.1 Installation    
10 Trouble Shooting
10.1 Network
10.1.1 Problems with routing
10.1.2 Bridging vs. Routing
10.2 Radio
10.2.1 Mutual Interference
10.2.2 Radio System Monitoring
11 VoIP equipment installation and configuration
12 R&D
13 Training
13.1 Radio
13.2 Flytech
13.3 Additional training
14 Recommendations
14.1 Radio
14.1.1 Backbone cost trade off study
14.1.2 Repeaters and CPEs
14.1.3 Monitoring
14.2 VoIP
14.3 Billing
14.4 Software
14.4.1 General
14.4.2 Specific features that are important in a wireless system
14.4.3 Specific features that are important in a VoIP system
14.5 Internet access test sites
14.5.1 Solar powered sites
14.6 Lightning protection
14.7 Mounting
14.8 Local construction
14.9 Transport
14.10 Further research
15 Conclusion
15.1 Concluding remarks by local counterpart

A Annexes
A.1 LVD Circuits
A.2 Battery charging and desulfation
A.3 Site Photos
A.4 List of Figures
A.5 List of Tables
A.6 List of Acronyms
R References
R.1 General
R.2 Batteries
R.3 Solar
R.4 Wireless
R.5 Lightning
R.6 Timers
R.7 SBCs
R.8 Other Wireless Projects


1 Background

Good communication services and  universal access are necessary for a higher standard of living and economic growth. However the high cost of legacy PSTN equipment may not be affordable to some developing nations, especially in rural areas which have a much lower subscriber density, or areas with geographic challenges such as large bodies of water, jungles, mountainous terrain etc.

1.1 Voice Communications

There are several paradigm shifts happening in todays telephony markets which are driving costs down by orders of magnitude. First legacy telephony systems are based on Circuit Switched Networks or (CSNs) This means a telephone call is allocated a dedicated circuit from end to end. In the old days this meant a physical pair of wires for the audio to travel over. Today this typically means two 64Kbps channels one in each direction which are dedicated to that call even if no one is talking, and since usually only one person is talking at a time about half of the bandwidth is wasted. For example, a typical small PSTN trunk can carry 24 or 30 simultaneous calls. If the bandwidth were used more effectively the circuit could carry much more if not almost twice as many calls. On the positive side CSN technology is very robust and predictable which made it easier to build reliable telephone networks in the early years of the industry. Because these PSTN switching systems were very big and centralized due to the state of the art at that time, they were very expensive and relatively few were sold to big companies like AT&T. So the market never developed to a point where the prices could drop significantly.

When computer networking technology was developed it was based on Packet Switched Networks (PSNs). Instead of dedicating a circuit of a predetermined bandwidth to two endpoints, packets are sent with little messages inside as each party has something to convey. This utilizes the bandwidth much more effectively. Instead of slicing it up into little pieces that are reserved but not being used half of the time, it is all consolidated in one big pipe that is only used when data actually needs to be sent. As the computer revolution evolved and the Internet grew exponentially so did the market for PSN technology products. This caused prices to fall by orders of magnitude.

Another Paradigm shift that is in progress now is sending audio on a data network rather than sending data on an audio network. Using modems over a Legacy PSTN is an example of the latter. This is a very unfavorable combination  because the modems at both ends usually send a carrier signal even when they have no data to send, and even though the PSTN eventually digitizes the audio it knows nothing about the data encoded by the modem so both 64Kbps channels are constantly in use even though the modems are not sending any packets.

The phone companies saw that this wasn't working very well and that there was a demand for lower cost data circuits, and started providing services like ISDN, Frame relay, and eventually DSL, and ADSL.  These services were designed to let the PSTN handle the communications as data and not audio. But ISDN still used a dedicated 64Kbps or 128Kbps channel so this approach did not capture a large portion of the achievable efficiency. The others had a quality of service metric known as Committed Information Rate or (CIR) which was usually set lower than the maximum bit rate of the Circuit, and paved the way for the consolidation of circuits into one pipe. These were some of the first steps taken in the transition from CSNs to PSNs in Telecom networks.

As high speed wide area networks (WANs) became more affordable and Voice over IP technology developed to become a commercial product thanks to standards organizations like the ITU and IETF, more and more organizations started buying high speed data connections between their offices and providing data and inter-office phone service over these links. Also many Internet Telephony Service Providers (ITSPs) started selling low cost long distance service over the Internet.

One limitation of this technology that may slow down the complete conversion to an audio over data network is that there needs to be power at the subscribers site for the terminal equipment. Legacy telephones are powered only by the PSTN so they will still work if there is  a power failure, and this is often when it's  needed the most. The PSTN is able to provide this by having a battery bank and generator at each switching site. To provide a reliable VoIP system it is usually necessary to have battery backup at each subscriber site.

Telephony equipment manufacturers could no longer ignore the compelling nature of these new communications paradigms, and now no one is building big switches anymore. Most of the new telephony products are based on PC platforms with Compact PCI cards at this time.

1.2 Data Communications

Data rates on wired networks have been increasing by powers of ten over the years, and more recently wireless rates have been catching up. This is due to many factors. Among them are the commercialization of spread spectrum technology, improvements in IC manufacturing processes to fit these radios on  small cards, and the allocation of radio spectrum in the Gigahertz range for licensed and unlicensed use of these devices.

The advantages of wireless networking are hard to ignore and the market for wireless Network Interface Cards (NICs) grew rapidly. Soon they too became commodity items. Initially they were targeted at networks within office buildings and homes, but many users found that they could also be used for long distance communications if the systems were designed properly. This use also became popular and another market grew which provided low cost high quality antennas and amplifiers to increase the range.

This became a low cost alternative to the Microwave links used by the Telecom and broadcast industries, though at not quite the same level of performance. Currently there are products available that work at 11, 45, 100, and 1000 Megabits per second. Though one should note that the expected throughput will be about half of the data rate, and generally the higher the speed the shorter the usable range.

1.3 Combining Wireless and VoIP

Wireless telephony is nothing new, there are microwave links for the trunk lines and Wireless Local Loop (WLL) for the subscriber terminal equipment. But it's mostly CSN based technology and is therefore quite expensive. If one combines a network built out of commodity wireless cards with Voice over IP equipment it is a low cost delivery infrastructure that makes efficient use of the bandwidth it provides. Additionally one gets a high speed data network that can also provide Internet access.


Of course its not quite that easy. Each radio repeater needs battery backup, and as mentioned before so does each subscriber. Also because of the change from the CSN model  to PSN it will be necessary to manage the bandwidth usage so that priority is given to voice traffic and that too many calls are not allowed to be placed simultaneously. This was not an issue with the legacy CSN systems because there were only a finite number of slots on the trunks for calls and when they were out of slots one got an all circuits are busy message.

Overall rapid growth in this area is expected, driven by fierce competition in long distance rates, and the large populations of people currently without good communications services.

2 Goals

2.1 Objectives

2.2 Expected outputs

3 Preliminary work

In the spring of 2001 the Consultant came to Bhutan as a UNV specialist attached to the Department of Information Technology (DIT). The Consultant demonstrated some wireless gear with a usable range of about 8km from a PCMCIA card in a laptop to a repeater and a similar range for some VoIP wireless phones. A proposal for this pilot project was drafted which can be found here [50]. Some more research was done on the equipment list and pricing, and in August the funding for the project came through. The Consultant returned to Bhutan in the spring of 2002 to complete the project.

4 Project Sites

The project has two locations, one in Limukha, and the other in Gelephu serving a total of about 80 customers. The project was intended to test the technology under different conditions. Limukha is more mountainous and Gelephu is flatter but has much more rain and lightning. The Network Operations Center (NOC) for the project was the main switching room of BT in Thimphu. Here were located all the servers, the gatekeeper, and a 30 line gateway which connected the VoIP system to the PSTN. Existing microwave links were used to provide network connections to the remote sites. This was much cheaper than building dedicated backbones, and fortunately spare E1 slots were available running to each area. Here are the site block diagrams:

Limukha Site Diagram
Fig. 1 Limukha Area Diagram.
Source: Tensin Tobgyl, ITU, "Bhutan: Wireless IP based Rural Access Pilot Project" [2.1]

In the Limukha area shown in figure 1 the E1 link ends at Dulchula and the last hop is done over a prototype backbone link which brings the network to Limukha hilltop. Don't be confused by the dish icons in the diagram they really do point at each other. On Limukha hilltop there is also an Omni repeater which serves three Customer Premise Equipment locations (CPEs). Talo Ridge separates Limukha from three other CPEs so we have the Talo omni repeater to cover those.

Gelephu Area Diagram
Fig. 2 Gelephu Area Diagram
Source: Tensin Tobgyl, ITU, "Bhutan: Wireless IP based Rural Access Pilot Project" [2.1]

In The Gelephu area, (figure 2) The E1 link terminates at the PSTN switching room in Downtown Gelephu. The Microwave tower there was used to mount the equipment to cover the surrounding areas. Because the design was very conservative two dishes were used to reach the outlining areas and one omni to reach closer CPEs. Happily it turns out that the omni was able to also reach one of the outlying areas. In each of these three areas there is a repeater to serve the local CPEs.

In this project each CPE  provides either four or eight analog phone lines. The Limukha area has six CPEs serving about 36 customers. The Gelephu area has Eight CPEs serving about 44 customers.

4.1 General guidelines for site layout

The constraints involved designing a high speed data radio system are very similar to ones in a WLL or cellular system. Everything needs to be Line Of Sight (LOS). This actually means a path which is also free of nearby obstructions such as corners of buildings and rooftops, not simply being able to see the other antenna. This is because there is something called the Fresnel zone around the centerline of the LOS path. Objects in this zone are likely to refract some of the signal toward the antenna and cause it to be attenuated. Of course sometimes one can get by with things in the way like a few wispy trees etc. But it's not a good idea and when the leaves get wet they will adsorb even more of the signal.

The range one can get out of each link depends on several things. The most important is probably the chosen frequency band. Because this project focused on a solution using commodity wireless hardware, this meant one of the ISM bands. Namely 2.4GHz or 5.8GHz. The trade off here is the higher frequency allow for higher data rates but shorter ranges, and more rain fade. So more power will be necessary to make up for the loss of signal strength when it rains or snows. At the time of this report 802.11b devices worked in the 2.4GHz band and  provided data rates up to 11Mbps, and 802.11a and others worked in the 5.8GHz band and provided rates up to 50 and 100Mbps. 802.11b equipment was chosen because of the lower cost and higher availability. For a quick introduction to 802.11b please see [20.5].

Once the band has been selected, the other factors influencing range can be adjusted: output power, receiver sensitivity, antenna gain, and data rate. Increasing any of the first three or decreasing the data rate will cause the expected range to increase. It is important  to also allow some margin for rain fade. The Project chose to put amplifiers on the repeaters and slightly higher gain antennas on the CPEs. A conservative rule of thumb for range is to try to limit the distance between repeaters and CPEs to about 8km. It was beyond the scope of this Project to explore the maximum distance between the repeaters but most manufacturers publish sample performance data for different configurations. A conservative estimate would be about 12-15km between 8dBi omnis and about 25-30km between 24dBi dishes using one watt amps. Of course the regulations concerning transmitter power, EIRP, and antenna gain will vary from country to country. There is a good discussion of the US regulations here [20.2].

4.2 Backbone

To build a full network a backbone is needed to deliver the bandwidth to the clusters of customer sites. Ideally the backbone should be much faster than the last mile delivery system so that many sites can be aggregated onto it for transshipment around hither and yon. Also because the backbone is a point to point system, one could take advantage of this and design it to be full duplex. This would more than double it's capacity, and lessen delayed packets due to collisions. This in turn would allow the maximum transit time of packets from one end of the network to the other to be much more predictable which is a consideration for VoIP and other real time data.

In this phase of the project there wasn't enough time or budget to explore a higher speed or full duplex backbone. It's interesting to note that as the number of calls in progress went up so did the collisions and retries. This is very understandable because as was mentioned before a call sends data in both directions and on a half-duplex link this means the two ends have to take turns sending on their shared frequency. Because 802.11b provides so much more bandwidth than is used for a moderate number of simultaneous calls, the collision rate is acceptable and the voice quality should be unaffected. On a system intended to run at near capacity one should seriously consider a full duplex high speed backbone. The most likely candidates seem to be 5.8GHz equipment with amps, perhaps on non-overlapping frequencies using horizontal and vertical polarizations for further isolation.

4.3 Last mile

The last mile delivery is typically structured with one or more repeaters serving the surrounding customers which need LOS or Near LOS (NLOS) to a repeater. In order to account for rain fade and get better range each repeater in the system has a one watt amp to boost the transmit and receive signals. For the repeater antennas 8dBi omnis were chosen. These seemed to be a good balance between gain and a radiation pattern which wasn't too flat. This also provided service to customers who were below the antenna at about a 30 or 40 degree angle. For the Customer sites 13dBi Yagi antennas were used since they were always served by one repeater and it wouldn't have been cost effective to put amps at each customer site. In cases where there is one CPE site way off by itself, it would be preferable to use an existing repeater if possible. In this situation adding an amp to a CPE site would be a viable solution. There are variations on this scheme where the CPE sites all talk to each other using 802.11b "ad hoc" mode and or a meshing protocol, but the available bandwidth typically goes down quite a bit and eventually a repeater will be needed somewhere to get back to the backbone. One should also consider how much bandwidth a community needs when choosing the last mile delivery technology. 802.11a and others can provide upwards of 25Mbps but as mentioned before the range is less. Since there are usually several non-overlapping frequencies available [17], either technology can be scaled up to easily triple the aggregate bandwidth in an area. Most customers who need a telephone line and perhaps an Internet connection can easily be served by 802.11b. Some organizations like hospitals and large government offices might require the higher rates available in the 5.8GHz band and the two delivery systems could compliment each other in these areas.

IEEE 802 Wireless WGs

802.11
 802.15
 802.16
 802.XY
Spectrum
 Unlicensed
 Unlicensed
 Licensed
Unlicensed

 Licensed
Freq Bands
 2 Ghz
 Various depending
on application
 10-66 Ghz
2-11 Ghz
 450 Mhz - 3Ghz
Range
 Local Area
 Personal Space
 Metropolitan
Area Access
 Metropolitan
Area Access
Mobility Support
 Portability
Local Roaming
 Personal Space
Connector
Avoidance
 Fixed
 Vehicular Speed
Mobility
Inter-Metro
Roaming
Station Power
 Battery
 Battery
 Mains
 Battery
LOS/NLOS
 NLOS
 NLOS
 LOS (10-66 Ghz)
NLOS (2-11 Ghz)
 NLOS
Group Charter
 PHY and MAC
for LAN
 PHY and MAC
for PAN
 PHY and MAC
for Fixed Pt.-Mpt.
Wireless Access
 PHY and MAC
for Vehicular
Speed Mobile
Access Networks
A quick comparison of existing and proposed 802 wireless standards [18]
Table 1

There is a lot of market pressure to provide a wireless broadband solution as can be seen by the many new proposed standards. See table 1 above for a few, such as 802.16 [19], or 802.XY [18]. Recently the NY Times reported that some entrepreneurs say they have solved the LOS restriction for 80.11b [20]. Most likely future projects will be able to take advantage of these new technologies as the bandwidth needs of rural areas increases.

4.4 Network layout
Fig. 3 Original network diagram for the NOC and outward links

At the NOC in Thimphu were the VocalTec servers, and Flytech routers all on one subnet. Figure 3  shows how it was initially set up, with photos in figure 4. Note that in figures 3, and 18 there is only one tower at each site even though there may be multiple tower icons. The network manager centralizes the management functions of for the VoIP network. The Gatekeeper controls the placing, routing, and logging of calls. The Real Time Server (RTS) logs the calls in real time, also known as Call Data Records (CDRs), and updates customer accounts. The billing server takes the account information and generates statements. The RAID array stores the Data Bases. The PSTN gateway connects via an E1 line to the PSTN using the R2 protocol. An E1 can carry 30 calls at one time so 30 of the approximately 80 customers could call numbers on the PSTN side of things at once. Additionally almost any number of calls to other VoIP phones could be happening simultaneously. Of course statistically only about 10% to 20% of a population will be using their phones at any given time. That is unless you have a high proportion of teenage girls.
a) First rack
 b) Second rack
Fig. 4 NOC in the Thimphu switching room

Dochula is the last mountain pass on the way to Limukha, and where the prototype backbone link was tested. Since it was only one hop to Limukha this could be considered a partial test. This was a point to point link configured with it's own SSID, on a different frequency from the Limukha omni, and this allowed it to run with little interference from other segments. At Limukha was the radio for the other end of the backbone and one to serve the Limukha area. They were connected together by a 10BT crossover cable.

Recall in Gelephu there were three radios intended to serve two outlying areas, and a closer one. These were set up as root radios [1] on different frequencies. All had the same SSID though one could force clients onto a particular radio if they were assigned different SSIDs.

These were the main repeaters for the project. There were also secondary repeaters to extend coverage over ridges and into low lying areas. The two project sites each had a large subnet that was further divided down into smaller subnets containing each repeater and the CPEs that they served. This helped organize things a bit.

5 Equipment choices and constraints

5.1 Radio

5.1.1 APs/Bridges
Recall that the 802.11b standard was chosen for the Project's radio gear. Initially the plan was to build the radio networks in each area from products of different 802.11b manufacturers but there didn't seem to be enough time or energy to do the additional research. Also the two top contenders Cisco/Aironet [20.8] and Orinoco/Wavelan [20.9] were very similar in price and performance according to the benchmarks so it was decided to simplify things a bit and just go with what seemed likely to work the best. Each manufacturer seems to have their own terminology for things and what the 802.11 industry calls an Access Point (AP) Cisco/Aironet calls a bridge. Probably because the Cisco equipment also acts as a network bridge. By the way, Orinoco/Wavelan calls their's Outdoor routers. At the time of this project Cisco had two product lines that were considered. The older 340 series which are basically the Aironet products unchanged and the newer 350 series. For the repeaters BR342s were used which have 100mw output, and have come down in price slightly since the 350 series came out. One should note that there is quite a lot of variability in the 802.11b PCMCIA cards on the market. Most are only 30mw output as are the 340 series cards. However the Cisco 350 series PCMCIA cards are one of the few with 100mw output so a few of these were purchased for experimenting with and site surveys. Recently 200mw cards have also became available.

If one looks at the 802.11 products closely they will notice that most of them either are just a PCMCIA card, are an adapter from a PCMCIA card to another form factor, or are a single board computer with one or two PCMCIA cards in it. Recently a few other form factors have come to the market namely Compact Flash (CF) and Mini PCI (PCI). From this it's clear that the radio components are always packaged in a popular small card standard and additional functionality is built around it. When comparing the price/performance of different products it's good to keep in mind the hardware/software costs. For example 802.11 PCMCIA cards vary from $50 to $100 or so, and Single Board Computers (SBCs) can be mass produced for $150 plus or minus. So a low end AP should cost about $200 and they do. But compare that to a high end unit like the Cisco BR342 or Orinoco Outdoor router and one wonders why the cost is on the order of $1000, many times higher. The answer is software and the companies good name. Orinoco makes this obvious by having several products that use the same hardware platform which can accept one or two of their wireless cards and are distinguished by the price / feature set of different firmware versions.

A more cost effective and flexible solution would be to take advantage of the new SBCs that are coming out on the market like the Soekris [42], and Musenki [41] SBCs. Populated with the wireless cards of one's choice they can be configured for virtually any situation. When a new wireless card becomes available just download the drivers and try it. One is not locked into a particular frequency band, data rate or vendor. There is a broad spectrum of free software available for these boards, with contributions from many people all over the world. One is not limited to the feature set offered by just one company.

Additionally this supports a philosophy of using a small number of modules to be used in different configurations depending on what's needed at each site. This way the inventory of different spare parts can be reduced. For example PCMCIA cards can be used for repeaters, CPEs, or laptops. SBCs can be used for repeaters, CPEs, VoIP gateways, or wired E1 routers. All using very similar software.
5.1.2 Wireless Ethernet converter
Because the VocalTec equipment was not wireless, each CPE also needed a wireless client adapter. Commonly these are PCMCIA cards but a stand alone device was called for here. The Avaya Wireless Ethernet Converter (EC) [20.3] was used and worked well for this pilot project, though it had some shortcomings. Originally all CPE sites were to have just one GW each. However for various reasons a couple of sites got rearranged or consolidated so that there were two GWs at these sites. The Avaya ECs could only handle just one network device on the wired side so this new arrangement didn't work. Apparently there is a Linksys product that will do full bridging [20.4],  but is not clear weather the AP also has to be a Linksys or not.

The other missing features for these products are that there is seems to be no way to monitor signal strength or to use SNMP with them.
5.1.3 Amplifiers
Because 802.11 is simplex system, ie. the radios transmit and receive with the same antenna on the same frequency,  when using amplifiers they need to be bidirectional. These will amplify the receive signal as well as the transmit, but they cost quite a bit more than unidirectional amplifiers which can be used on full duplex links. There are several vendors of bidirectional amps, Hyperlink [20.6]and YDI [20.7] are two popular companies and their amplifiers seem to be similar in price / performance though the Consultant doesn't have much experience with YDI. Hyperlink also sells antenna kits with their amps which are competitively priced. The 802.11 repeaters used the Hyperlink 2.4GHz one watt amps.

Full duplex links typically use separate transmitters and receivers each with their own antenna. If a full duplex link were to use amps then then a transmitter  would use one to amplify the transmitted signal, and a receiver would use a different type of amp to amplify the received signal. These amplifiers are cheaper to build because they only amplify in one direction.
5.1.4 Antennas
Antennas vary in price quite a bit, and one can even build their own without too much trouble. Hyperlink has a good selection of moderately priced antennas, and will give one a discount if they order a kit. The Omnis and Yagis are about $100 and the dishes have come down a bit to about $70. Most of the 802.11 antennas on the market are readily available with weather proof plastic enclosures (radomes ) except it seems, the dishes. This makes it rather expensive to design backbone links where there is snow and ice. For CPE antennas next time the Consultant would recommend Yagis from this South African company [5.1]they seem very solid and weather proof and are only $28[5.2].

5.2 Mounting hardware

It's helpful if all the mounting hardware uses readily available nuts and washers because one will lose more than one cares to admit by dropping them off of towers or roof tops. Also u-bolts can be easily made from long pieces of threaded rod sometimes called all-thread. It's good to keep a supply of several sizes handy along with a box or two of nuts and washers. Also it can be used for mounting to concrete footings.
a) Cracking when bolt is tightened
 b) Hard to find proper sizes in the field
Fig. 5 Problems with expansion bolts

Figure 5  shows some examples of the problems one can encounter fastening to concrete footings. The intention of course is to pore the concrete with the bolts in place so the installing crew can come along later and bolt down the equipment mounts. Usually this requires too much coordination between the two crews so the concrete is pored without bolts. Then the installing crew comes later and drills the holes. So far expansion bolts have been used but as one can see in figure 5 that there can be problems with the concrete cracking when the collar expands, or just finding the correct sizes of bolts, expansion collars, and washers can be tough when supplies are running low. The Consultant recommends using all-thread and epoxy anchoring compound [5.6]. This way the same supplies of all-thread, nuts and washers can be used for anchors as well as u-bolts.

5.3 Supplies

It's important to waterproof cable connections properly. One way is with special rubber tape that self seals to itself once applied. It's very important to cover the rubber tape with a UV and weather resistant electrical tape because the rubber tapes will eventually crack and breakdown with prolonged exposure to sunlight. See figure 6 for examples.

a) Example of coax entering box
 b) Electrical tape being applied over rubber tape
Fig. 6 Proper taping of connections		 c) Example of taped Amp connections

One can also use a brush applied waterproofing compound such as ScotchKote [5.7].

Another handy item is epoxy putty for sealing holes in roofs and the like. The common brand name in Bhutan is M-seal. Of course wire ties are a good way to dress cables but don't use them on lightning ground wires because they will melt. It's better to use stainless steel straps instead.

5.4 Linux Routers

The thought here was to demonstrate the viability of open systems and especially open source software solutions as alternatives to more expensive commercial products. Briefly, the Consultant put together very small computers with E1 cards running Linux which served as routers at the ends of the microwave E1 links. For more details on their configuration and maintenance please see the Flytech training manual [2].

The Flytech boxes did meet most of these expectations. They excelled at being flexible and provided a work around for an apparent problem with the Cisco Bridges. Also they were much more capable than a mere router needed to be, which was a double edged sword. On one hand it allowed network monitoring and logging to be done from one box (See section 10.2.2), on the other hand the system image was quite large and this made it a bit awkward to back them up. Because they were essentially small PCs they suffered from some of the same limitations, such as a risk of file system corruption when there was a power failure, and the possibility of hard drive failure. Because they are so flexible they may require a little more skill to administer if one is contemplating adding features. Though for the usual additions to the network, typically only one file needs to be edited.

If one agrees that this is a good direction to go and that the flexibility gained was a factor in the success of this project then most of these issues can be addressed. The possibility of file system corruption can be almost completely eliminated by using a journaling file system. The Consultant has done this on other Flytechs and no longer has to worry about power interruptions. The Hard drive can be replaced with a compact flash card since the Flytech's have a CF slot. The tradeoff here is that there is limited space for additional applications and no space for logging large amounts of data. On the positive side back up is a breeze, and administration tends to be simpler because the system is smaller and more focused on one task. For sites with battery power there is also a DC supply option for the Flytechs.

All in all the Flytechs did their job well, but they were probably overkill. The cost for a Flytech was about $2000 total, comparable to a Cisco router which is much less capable. Though if the system monitoring is to be done from another server then the routers could be scaled down somewhat with a similar reduction in cost. The Soekris SBCs might be able to replace the Flytechs someday when an E1 card is available for their Mini PCI slot. Soekris has plans to come out with such a card but the timing isn't certain. This configuration might cost around $500 to $1000 USD but that's just a guess.

5.5 VoIP

For the VoIP component of the system there are a lot of considerations but due to time constraints the Consultant feels that some were left unaddressed. Still a reasonable choice was made that worked adequately. Initially there was interest in wireless VoIP products, and at the time of the research there were only one or two on the market. Primarily the search was for a fixed wireless VoIP residential gateway rather than mobile wireless VoIP phone. E-tel provides the former in their GW210 model [4], and Symbol's NetVision phone [5] is an example of the latter.

The Etel GW210,  even though it was only a two port unit, was felt to be cost effective because it combined the VoIP and wireless components of a CPE into one unit. However it was difficult to confirm that it would interoperate with any of the gatekeepers (GK) that the project was looking at. In hindsight E-tel's list of GKs that were compatible with the GW210 could also have been checked into and then billing systems that worked with those could have been found.

 Eventually VocalTec [3] was chosen as the supplier of the VoIP equipment for several reasons. A complete solution with a billing system was needed. They were very helpful, and in fact they were one of the few vendors that returned phone calls. Also the project had the usual time constraints and they seemed to be the only choice when it was time to make one. VocalTec was unable to interoperate with third party gateways (GWs) so other products could no longer be considered, but this also had it's advantages because the two viable models had four and eight ports, compared to most (but not all) of the others that had been looked at which were one or two port units. This allowed the CPEs to be consolidated so they served a small cluster of buildings with one antenna. This is preferable to many CPEs and antennas because there is less contention for the radio channel.
5.5.1 Billing
VocalTec recommended the Mind billing system, a third party product which seems to be working well. Again if there had been more time there was already an in house billing system in use by the ISP branch of Bhutan Telecom (BT) and it might have been possible to adapt that one to the VoIP project. Especially since there were local programmers who maintained it.

5.6 Power

As noted above, one of the other differences between a legacy telephone system and a VoIP system is that the CPE must be powered locally at the Customers site. This is because there are no wires to carry the power. Now if one were considering a wired VoIP system using 10/100BT then they could take advantage of a new development called Power over Ethernet (PoE) which runs power over the Cat 5 Ethernet cable. In this case though the power had to be provided locally, so instead of a centralized and easily manageable bank of batteries at the Switching center one needs batteries at each customer site along with a charger and Low Voltage Disconnect (LVD).
5.6.1 Commercial
At sites that had commercial power a battery charger that had a built in LVD was used . Apparently there weren't a lot of choices locally  for this product and the one that was available was not a great match for this application. The unit was larger and heavier than it had to be (figure 7).
Fig. 7 The commercial charger was a bit larger than it needed to be

Also it seemed impossible to get documentation for it, and it appeared to be malfunctioning. Eventually an unlabeled LVD adjustment was discovered (Fig 8 d) and, that it was not set properly at the factory. In fact they seemed to be set randomly on each unit. So trips had to be made back out to the CPE sites and readjust the LVD set points. This was after a couple of batteries were damaged.
a) Top of Charger
 b) Back of Charger


c) Right side of Charger
 d) Closeup of LVD board showing adjustment
Fig. 8 Inside the commercial charger

The lesson here is carefully research the products for one's system. Even though it is better to buy locally it still may not be the wisest thing to do.
5.6.2 Solar
For sites where commercial power was unavailable solar panels and charge controllers were used. There were both repeater and CPE sites which used solar power. The panels were purchased from Tata BP solar and were 70 watts each.

The Consultant recommended charge controllers with integrated LVDs, and suggested several such solar chargers during the selection process [14]. But apparently there was a lot of pressure to get the project underway and not enough time to evaluate them all. An outside vendor recommended the Trace C35 and C40 thinking that they also had LVD options, This is an understandable mistake because they can indeed be used as an LVD but not when they are configured as solar charge controllers. Also the specifications can sometimes be ambiguous, when in doubt it's best to contact the manufacturer. This was still an excellent choice because trace produces some of the highest quality alternative energy products on the market. Eventually a standalone LVD was found to work with the C35 but it's quality was suspect.

As these examples show it can sometimes be difficult to find units correctly sized for a site with all the desired features. In this case building one's system out of several smaller units which better fit the constraints is probably advisable. In this instance the high cost of quality LVDs [15], or the low quality of locally produced units seemed to be an issue. The Consultant eventually did further research and designed a very low cost (VLC) LVD which could be used for any repeater or CPE site. The parts cost is around 10 to 15  USD. See Appendix A.1 for details.

There were several solar powered CPE sites which required much less power than the repeaters, and it seems there are quite a few small charge controllers with LVDs available. Though the Specifications are not completely clear on this matter the Trace C12 seems to be LVD capable when it is configured as a charge controller unlike the C35 and C40. In any case the reliability of future CPE sites would benefit by having integrated Charge controllers, and LVDs.

Also note that when the sun is shining that no part of a solar panel should be shaded. This is because a great number of cells in a panel are connected in series, and that when shaded, a cell will act like a resistor. This will greatly reduce the output of a panel even if only a corner is shaded.
5.6.2.1 Sizing of solar systems
The solar repeater sites were sized such that they could operate from batteries for eight cloudy days in a row and be recharged in three. During cloudy days the panels will only put out about 10% of normal but to be conservative it is assumed to be zero. To calculate the size of the solar array and battery bank it helps to make some approximations. First the output of the solar panels need to be averaged or "derated" over a whole day because they will only give their rated output during high noon and if they are not too hot. In fact many solar manufacturers rate their panels at 25 Degrees C but they are usually much warmer and so put out less power. Also it is convenient to talk in terms of amps and amp hours even though batteries change voltage with their State Of Charge (SOC) and the number of watts per amp hour will actually vary.

For the calculations one starts with the power draw of the load, for the repeaters that was usually 18 Watts.  Then one needs to estimate how many hours of direct sun light the panels will  usually get on a sunny day for a given site. Six were used here, please see table 2 for example calculations.

1)
First note the size and deratings of the panels and batteries.
Panel wattage = 70W
 Panel volts = 17V
 Derating = 95%
Battery Ah = 100
 Maximum usable = 80%
 Charging efficiency = 75%
2)
 Next get the average current draw for the load,

Current = Load Watts / Volts
 = 18W / 12V = 1.5A
3)
 Then calculate how many amp hours the load consumes in a day.

Total load per day = 24 hours * Load Amps
 = 24 hours * 1.5A = 36Ah/day
4)
 Also during the eight day autonomy period. This gives the needed battery bank capacity.

Days autonomy * Amp hours/day = Total amp hours for autonomy period.
 = 8 days * 36Ah/day = 288Ah
5)
 Next one needs to realize that the batteries can't be discharged completely
because it will cut their expected lifetime almost in half.

Amp hours of autonomy / Percent usable = Battery bank size.
 = 288Ah / 0.80 = 360Ah
6)
 Then the fact that one will always put more back into a battery than
one can get out of it needs to be accounted for. This is the expected
charging efficiency (or inefficiency).

Amp hours of autonomy / Charging efficiency = Amp hours to put back

So about 384 Amp hours needs to be put back into the battery bank
after a full eight days of cloudy weather.
 = 288Ah /  0.75 = 384Ah
7)
 Now find out how many amps are needed during the sunny part of the day.

Total recharge Ah / Days for recharge / Hours of sun per day = Total Amps during sun
 = 384Ah / 3 days / 6/day = 21.3A
8)
 Next find out how many amps the load will need during the
sunny part of the day to break even with the battery drain at night.

Load Ah per day / Hours of sun per day / Charging efficiency = Load sun amps.

Note this is a short cut that made it seem a little worse than it
was because during the sunny part of the day the load is powered directly
off the panels and the battery inefficiency doesn't come into play,
but the result wouldn't change very much and it is better to
overestimate ones power usage a bit.
 = 36Ah / 6 Hours / 0.75 = 8A
9)
 Now add the current needed to recharge the batteries with  what's
needed to power the load at night which will yield the total current
required in sunny weather.

Recharging current during sun + Load current during sun = Total current needed during sun
 = 21.3A + 8A = 29.3A
10)
 Then get the average amps each panel can be expected to produce.

Panel Derating * Watts / Volts
= 0.95 * 70W / 17V = 3.91A
11)
 Now one can easily calculate the number of panels needed.

Total amps / Amps per panel  = Rounds up to number of panels.
 = 29.3A / 3.91A/ Panel = 8 Panels
12)
 Last but not least do the same for the battery bank.

Battery bank size / Each battery  =  Rounds up to number of Batteries.
 =  360Ah / 100Ah/Battery = 4 Batteries
Calculations for solar system size
Table 2

In some situations one could save some money by installing a Maximum Power Point Tracker (MPPT) which allows the panels to supply current at their most effective operating voltage rather than whatever voltage the batteries happen to be at. There are claims that these devices can get an additional 25% to 30% out of the panels but this is often not realizable. Still in certain cases they are warranted [13]. If one used an MPPT as a charge controller then the above calculations would be more accurate if they were converted to use Watts and Watt hours instead of Amps and Amp hours. One can get a hint of the difference this might make if the solar panel wattage is divided by the nominal battery voltage (12V) instead of the solar panel working voltage (17V) in step 10 above.

One MPPT on the market that seems to be sized right for the CPEs is the B.Z. products MPPT200 [13.1]. It should be seriously looked into as an alternative to the Trace C12 since it also has a built in LVD and is competitively priced.
5.6.2.2 Batteries
Lead acid batteries still appear to be the most cost effective solution for most situations though one should take into account the cost of an environmentally sound way of disposing of worn out batteries when making comparisons. Again in legacy Telecom systems all the batteries are at the local switching center where they are kept in a controlled environment and monitored regularly. This makes it feasible to use ordinary flooded cell lead acid batteries with Catalytic recombiners. On the other hand a VoIP system requires many batteries in the field where there is little control over operating conditions. Also they often need to be transported and handled by less experienced people. In this situation sealed lead acid batteries are the better choice even though they are more expensive. There are several types of sealed lead acid batteries, Valve Regulated (VRLA), and Gel-Cells are two examples. Gel-Cells won't spill if tipped over and can often be mounted in any position. Proper venting is also important, especially for flooded cell lead acid batteries. There have been accounts of roofs being blown off by hydrogen explosions.

In any case there will be a sizable investment in batteries for the system. Therefore one would want to maximize the lifetime of the batteries and minimize their maintenance requirements In order to lower the cost of ownership. This means extra care should be taken when choosing a charging / monitoring system. Again all things being equal it is desirable to buy locally but carefully research the quality of all products before making a choice.

There is a wide spectrum of chargers available for lead acid batteries. Generally the higher end three stage chargers are required here, and ideally one with an integrated LVD. The LVD protects the battery(s) from being over discharged which would seriously damage them. In fact if a Lead Acid battery is only discharged to 80% of it's capacity then that will almost double its lifetime [9].

Also it's interesting to note that it gets harder and harder to put energy back into a battery as it gets fuller and fuller. This is called it's charging efficiency, and it varies nonlinearly with the battery's state of charge (SOC) [11]. The Consultant's hypothesis is as the battery is charged the chemical reactants are converted to the charged state. As the battery nears full charge less and less reactants are available for conversion, and the internal resistance of the battery goes up. In the end about half of the power going into charging the battery is wasted as heat. If this is not taken into account when designing solar systems the panels may be under sized and may not be able to keep the battery bank fully charged. Most likely  this is not an issue here because of the aggressive recharging time that was required for these solar systems. There are several good papers on the appropriate design of battery systems for solar sites at Sandia Labs [10].

5.7 Timers

As a bit of added insurance for the repeaters the Consultant felt that it wouldn't hurt to add a timer that would power cycle the unit once each day at four in the morning. Sometimes computer based devices (also known as embedded systems) can lock up for unknown reasons and then need to be rebooted. A timer is a low cost solution that can save one a trip into the field. If one had more control over the specification of the embedded systems then a watch dog timer is the preferable way to protect against these type of problems. Watch dog timers are built into a lot of modern microcontrollers. They act like a dead man switch and once activated they need to be "touched" by the software every half second or so or they cause the system to reboot. If properly implemented they add a good fail safe component to one's embedded system. For this project low cost digital timers were purchased from Amazon.com [34] and modified to work on 12V. There are also ready made 12V digital timers but they seem to be quite a bit more expensive, probably because there is not as much demand [35].

5.8 Weather proof enclosures

The repeater equipment needed to be protected from the elements and rather than shipping metal boxes from some faraway country a local supplier was sought for them. It turned out that one of BTs metal shops was able to make very nice weather tight boxes for about $20 each, compared to $40 from India, and $100 from the US. They even sported sharp hand painted logos!
a) Weather tight box made in Bhutan
 b) Slight problem with mounting
Fig. 9 Weather tight box

There were a few issues with mounting though because the Angle iron was upside down. See figure 9 b above. Certainly the next batch will come out just fine.

6 Grounding and Lightning Protection

There was a lot of concern over proper grounding and lightning protection, and for good reason. In Gelephu There are lightning storms several times a week in the summer. Sometimes At night it's as if someone were arc welding in the sky, and one can almost use that light to see by!

The Bhutanese have a lot of experience with lightning protection because after all they live in the land of the thunder dragon. Generally they use lightning rods with spikes and heavy metal straps on the roofs running to deep earthing pits. These pits are usually about 3 meters deep with a plate of copper in the bottom to which the grounding strap or wire is bolted. The pit is filled with layers of salt and charcoal both of which are conductive and help to retain moisture. The top meter or so is filled back in with dirt. In very dry areas an additional measure which helps is to leave a pipe in the ground which ends near the top layer of  salt and charcoal. A funnel can then be used to poor salt water back into the earthing pit [27]. This will keep the ground moist and replenish the salt content which dissipates over time. There are also other chemicals that will increase the conductivity of ones ground and last longer than salt [28].

At sites where only an equipment ground was needed copper stakes were used connected together by the grounding wire which then ran to the equipment. A earthing meter was used to measure the resulting ground to check if adequate.

Even so with all this experience and good practices there are still many thousands of dollars damage caused by lightning each year. For the project's relatively small sites there was some discussion as to what the most cost effective earthing practice would be.

What it boiled down to was that large voltage potential differences are to be avoided. Keep in mind the with even the best grounds the area surrounding a lightning strike will rise tens of thousands of volts for a few micro seconds. This includes all of ones equipment if lightning strikes the tower, it just can't be helped. But by itself a rise in potential doesn't cause damage. It's only if part of ones equipment is swinging to the tune of a different voltage that causes problems [23]. Single point grounds are the best way to prevent this from happening. This will also avoid ground loops which are always undesirable in electrical systems. Simply put, a single point ground is where each ground stake and each piece of equipment including the lightning rod have separate runs of grounding wire to a common point, usually a terminal strip or bus bar. If that sounds too expensive there are some shortcuts one can take that don't appreciably diminish the protection provided.
See figure 10.
Fig. 10 Different grounding examples

One should avoid at all costs inadvertently grounding different interconnected parts of a system to different grounding points. If there is a nearby lightning strike then most likely the different grounding points will develop high potential differences as the surge passes through them, and this will destroy the equipment. This is an example of how an improperly grounded site can actually cause the equipment to be damaged instead of protecting it. See figure 11 c.
Fig. 11 Equipment grounding examples

It's the Consultants opinion that there is a situation where two grounds are acceptable, as long as there is no electrical path from one one ground system to the other. Thus keeping them separate so that even though they may be at different voltage potentials no current flows between them. Since lightning is so unpredictable guaranteeing that no current will flow is difficult. It would probably be unwise to have two separate grounds in the same building because the strike could jump from one system to the other. On the other hand it might be best to give two towers 50 or more meters apart separate grounds. In this case one would use separate grounds and carefully ground cables that came in from outside the local grounding system of each tower. Commercial power is a good example of cables entering from the outside. It's common to put large Metal Oxide Varistors (MOVs) across terminals in the panel to protect against surges and some strikes.

For example there may be a microwave tower at a site but there are other antennas on it using the same band as the equipment. Then it may not be feasible to put one's radios on the same tower. If one erects a smaller pole or tower a hundred meters away then it may not be practical to ground the two towers together.

When grounding ones antenna or coax it is a good idea to have several loops of coax before it enters the equipment box or building. Because lightning causes such rapid surges even small inductances have very high impedances. This means lightning will tend to avoid going through coils and try to find a short straight path to earth.

For the repeater sites that were not colocated on preexisting towers, lightning rods were installed at the top of the poles with the antenna about a meter below. Additionally it's advisable to run the grounding wire for the lightning rod down the opposite side of the pole from the coax giving it the best chance to survive a strike.

providing single point grounds for solar repeater sites was easy. Sites with commercial power were harder because the power line was also grounded somewhere else. The neutral side of the power line isn't such a problem because it could go though ones local single point ground but the hot side could transmit any surge into the site. In the US the model for lightning induced surges on the power lines is a nanosecond rise time up to 6000 volts with a 20 microsecond decay.  It would be best to use a lightning arrestor or surge suppressor on these lines when they enter the electrical "panel" for a site.

Often lightning arrestors are of the gas discharge type, and these fire relatively slowly when compared to the rise time of a lightning strike which is on the order of nano seconds.  TransZorbs are another common surge suppressor based on controlled avalanche diodes with sub nano second response times [31].  These are often useful in protecting equipment on phone lines from surges caused by nearby lightning strikes which are then inductively coupled into the lines. ONEAC is another manufacturer of high quality power conditioning equipment and surge suppressors [32]. They also have a popular phone line suppressor [33].

For most of the CPE sites lightning rods were not required because the metal roofs acted as a shield. The Yagi antennas were installed just under the eves pointing out toward the local repeater. It would be extremely unlikely that lightning would curl under the roof to find the antenna. Most of these roofs didn't have lightning protection being village houses, but sometimes there were cables holding