- 2.1 General
- 2.2 Wrong kind of fiber
- 2.3 Currently installed fiber
- 2.4 Hubs are a security and performance problem
- 2.5 Performance
- 3.1 New fiber
- 3.2 Future network topology
- 3.3 Switches
- 4.1 Cable TV
- 4.2 Video surveillance
1. Introduction
This paper describes plan to upgrade TRINET [1] backbone to gigabit ethernet (GE) [2] and user connections to switched 10/100 Mbps ethernet. This means massive single mode fiber installation, different network topology, modular switches on every wiring closet and CAT 5 cabling on buildings. I won't be going to details as they change all the time. The big picture should be clear after reading this.2. Current situation
2.1 General
TRINET is growing fast in terms of the number of users and the amount of traffic. We have increased capacity by adding 10 Mbps segments so that most building have separate switched 10 Mbps connection. Those segments are connected to hubs in buildings and to a core switch. Segments are half duplex as they are connected to hubs. Larger segments do still have lots of collisions. Most of the traffic goes out so the old 80% inside and 20 % outside rule doesn't apply. It is more like 10 % inside and 90 % outside.Users are relatively pleased with current situation, but some buildings have so old equipment that administrators nightmare may come true any day. Our hardware is very different in different buildings. Some have basic hubs that are connected with thinlan. Others have eavesdrop preventive switching hubs that can separate four internal and four external collision domains. Management allows to isolate ports to different groups, follow traffic amounts etc. It is a lot easier to find faults in managed environments, BUT the problems may also be much more complicated.
2.2 Wrong kind of fiber
We do have several kilometers of fiber optic cable, but it is multi mode (GK). It carries GE up to 550 m but that is just not quite enough. Current topology is also a big problem. The cables go like a worm from one building to another. This means that we have plenty of fiber that can never be used. Lack of money and the price of fiber in the past has mostly dictated planning. Now we have different situation and solid intention to do it properly. Existing fiber network can in the future be used as a fallback option and backup link. Minor problem is also the variety of fiber optic connectors. We have all the possible models installed depending on the time of installation.2.3 Currently installed fiber
GK = multi mode fiber SM = single mode fiber
2.4 Hubs are a security and performance problem
All our users are connected to hubs. Hubs are a shared media meaning that all of the traffic is repeated to all stations connected to that segment. It makes it easy to eavesdrop traffic using simple unix tools such as tcpdump. This has been a serious problem in our network. In the beginning of 1997 we had two separate networks each having four class C networks joined together using CIDR [3]. Only few bridges divided massive segments from each other. Imagine the number of cleartext passwords passing through as both networks had about 400 users using telnet and ftp. SSH has now eased this problem, but not all users take these password matters seriously.Reliability and useability in this kind of network should be very high. It takes only one hub to fail and all the traffic in that segment may stop if the hub happens to be in suitable place. We have had cases where one port of the hub has broken down and the result has been massive number of collision on entire segment. Old hubs don't have any kind of management so very little is known about a certain ports traffic. Problems have to be solved onsite using trial and error mechanisms.
2.5 Performance
It is unusual to get better than 3 Mbps performance for a single user with our current network configuration. It is though much better than any internet operator can offer for home users. Utilization can be very high on certain time of the day. 100 Mbps backbone to HUT computing centre has never really been a limiting factor, yet. The reason is of course the fact that traffic get limited naturally very close to station. Collisions and the amount of traffic prohibit a single user to reach maximum 10 Mbps ethernet performance.Following statistics show the utilization of our 100 Mbps ethernet link to HUTCC.
Traffic Analysis for TRINET
The statistics were last updated Sunday, 3 May 1998 at 11:20`Daily' Graph (5 Minute Average)
| Max In: | 1127.6 kB/s (9.0%) | Average In: | 253.6 kB/s (2.0%) | Current In: | 235.2 kB/s (1.9%) | ||
| Max Out: | 1216.6 kB/s (9.7%) | Average Out: | 258.6 kB/s (2.1%) | Current Out: | 379.6 kB/s (3.0%) |
`Weekly' Graph (30 Minute Average)
| Max In: | 803.2 kB/s (6.4%) | Average In: | 289.8 kB/s (2.3%) | Current In: | 262.6 kB/s (2.1%) | ||
| Max Out: | 1124.7 kB/s (9.0%) | Average Out: | 284.2 kB/s (2.3%) | Current Out: | 222.5 kB/s (1.8%) |
`Monthly' Graph (2 Hour Average)
| Max In: | 777.0 kB/s (6.2%) | Average In: | 288.2 kB/s (2.3%) | Current In: | 181.7 kB/s (1.5%) | ||
| Max Out: | 2152.1 kB/s (17.2%) | Average Out: | 332.5 kB/s (2.7%) | Current Out: | 47.1 kB/s (0.4%) |
`Yearly' Graph (1 Day Average)
| Max In: | 438.5 kB/s (3.5%) | Average In: | 284.6 kB/s (2.3%) | Current In: | 216.3 kB/s (1.7%) | ||
| Max Out: | 486.8 kB/s (3.9%) | Average Out: | 329.5 kB/s (2.6%) | Current Out: | 178.9 kB/s (1.4%) |
| GREEN ### | Incoming Traffic in Bytes per Second |
| BLUE ### | Outgoing Traffic in Bytes per Second |
3. The plan
3.1 New fiber
Before we get to install the cables we have to prepare routes and canals. Different service tunnels and cable canals already exist, but hundreds of meters elastic pipe and cable shelve have to be installed and tens of diamond drill holes has to be done before we can even start. Planning and preparing routes carefully saves us from damaging cable during installation and the actual installation time gets shorter. We also have to consider the well being of the cable in coming years. Ice, mechanical contact, water everything has to be though about. Minimum bending radius has to kept in mind in every turn. Cable tolerates relatively much bending when it just lies on cable shelf, but when cable is being pulled on tight canal the power concentrating on small area can be huge.New fiber optic cables having 36 single mode fibers are installed is star topology. Every cable begins in Jämeräntaival 7 and ends in corresponding building wiring closet. Total cable length and costs increase, but this solution eases pigtail connector installation, improves reliability and decreases attenuation which is roughly 0,5 dB/km in fiber and 0,5 dB in on adapter. So one adapter corresponds 1 km fiber in sense of attenuation. Fibers are welded to pigtail-fibers that have factory installed SC UPC connectors on them. Connectors are inserted to fiber optic termination box having 18 duplex SC adapters. Last thing is to measure attenuation on every fiber and label connectors.
3.2 Future network topology
SM = single mode fiber
Future topology on map of Otaniemi
3.3 Switches
Switches on mapWe have planned that the core switch should have around 30 GE ports. Two or more of them would connect to HUTCC, Mediapoli etc. Routing has to be done in our end and that would be in the core switch. On top of routing cababilities the core switch should be able to priosize traffic depending on the tcp port. This feature comes handy when receiving video or similar time depenging data. The core switch does all the hard work.
All the edge switches are directly connected to the core switch using single mode fiber. Edge devices should have either stacking feature or they should be massive and modular. Largest wiring closet has over 200 connectors. There are very few options for this scale of demand of 10/100 ports. Smaller wiring closets can do with switch that has 48 ports and
4. By-products
4.1 Cable TV
It is not purely coincidence that Cable TV can also be transmitted using a singe mode fiber. We have been negotiating with several fiber optic cable TV equipment manufacturers, such as Hirschmann and Teleste, about the equipment to rebuild our cable TV network at the same time. This would be significant synergy benefit. Cable TV would use separate fiber (only one) to transmit analog TV and radio signal. Digital TV and radio are an option like return path. That's why two fibers are reserved for cable TV. In fiber Cable TV network attenuation isn't any more a problem at least in our distances. Building networks start to be limiting factor. This leads synergy benefits in recabling CAT 5 and antenna networks at the same time.4.2 Video surveillance
Bike and car thieves are some kind of problem in Student campus. Centrally managed video surveillance would be easy to set up to work over the network. Any thief would think twice if bike cellars had video cameras and time lapse VCR recording continuously.5. References
[1] Kyläverkko, Veijo, TRINET's home page http://www.tky.hut.fi/~verkko/[2] Gigabit ethernet alliance, Homepage http://www.gigabit-ethernet.org/
[3] Network Working Group, Classless Inter-Domain Routing (RFC 1519) ftp://ftp.funet.fi/pub/standards/RFC/rfc1519.txt
6. Interesting links
Cabletron SmartSwitch RouterThe switching book
Cisco LAN switches
Xylan products
Nbase switches
Bay Networks - Accelar routing switches
Extreme networks switches
This page is maintained by
Tommi
Saranpää, E-mail: tommi.saranpaa@hut.fi.
Updated 3.5.98
URL: http://www.hut.fi/u/axelf/tik-110.551/trinet.html