LEC Network Support
T1 and T3 connections come with different network support than residential DSL and cable internet connections; although strictly speaking everything here applies to a business using a DSL or cable internet connection too. There are some things that you should know that can make outage resolution quite a bit faster. For example most vendors have a web interface to their ticketing system which is quite a bit more convenient than constantly calling customer service. DSL and cable internet connections are less reliable and less frequently involve a technician being sent to the customer’s location, but it is a fairly high percentage of T1 and T3 issues that involve it. There are formal escalation procedures that can easily bring immediate attention to a neglected ticket if you know to use them. This page can help you to know the system to let it work for you rather than against you.
On T1 and T3 connections, the local exchange carrier (LEC), and sometimes the communications firm contracted for the service (if different), are able to test the circuit remotely when providing network support. There are two types of tests: intrusive and non-intrusive. Intrusive testing actually interrupts the service on the circuit while the circuit is tested for any issues with different types of diagnostics. Non-intrusive testing is simply where the circuit is monitored for whatever issues may exist. Both types of testing can be done automatically and manually. It is actually common that a circuit outage is resolved as a result of an intrusive test even if there is no human involvement in the testing. Non-intrusive testing does not usually complete with successfully correcting a circuit issue. If a circuit is down or degraded past the point of usability it is best practice to intrusively test the circuit. If the circuit is down already then it cannot hurt anything. Testing is normally the first thing that the provider will want to do when providing network support. If the circuit is degraded (taking errors or packet loss) it is wise to ask the provider to intrusively test the circuit after business hours within a test window of 4 hours or more. This author’s opinion is than non-intrusive testing is (usually) a waste of time. Cisco routers and switches both automatically provide error statistics on interfaces which is much more useful.
If you ever think that resolution is taking too long contact the provider’s support number and say “please give me a # level escalation”. An escalation is simply a formal notification that a ticket needs attention from the next logical person who can fix or help fix the issue. When an escalation is granted, the following information is recorded: the first and last name of the person escalated to, the job title, and the usually the company. Escalations start at level 1 and go upward in levels sequentially, If your support contact is a different firm from the LEC the LEC’s escalation is separate from the firm directly handling your ticket. Beware of escalations going to the network support agent currently handling the call or going to the company with first look rights when it’s more appropriate to go to the LEC. If you are waiting action from the LEC specify that the escalation be with them. Because it’s a formal process and it notifies the relevant person, it is highly effective in expediting resolution; much more impact in one sentence than an hour of arguing on the phone. Don’t ask for an escalation unless 60 minutes has passed since the previous escalation was generated and you think it could be handled faster. It isn’t supposed to be granted until at least 60 minutes from creation and 60 minutes after the previous escalation with most carriers. However when you do it there will be a lot less stress if you stop to think about the carrier’s side. What is currently happening on the ticket now? Is someone working on it? Is the ticket waiting information from the customer or sitting in a queue waiting for attention? If the vendor is legitimately working on it don’t ask for the escalation until a kick in their pants can actually help. You can only get an escalation once at each level and some providers deny escalations if they don’t feel it is justified. Wait until they are letting it sit, but carefully watch their progress to ensure it isn’t waiting anything from you and has progress; it probably has a serious business impact if people are dependant on it. If your only means of watching a ticket is calling them don’t be afraid of calling network support too frequently. Once an escalation has been made, give them the time they need to take action on it. If you’ve lost time with a mistake or non-action by the carrier’s network support team going ballistic isn’t going to help. They still need time to fix it.
It is very common for carriers to get tickets caused by remote power outages, configuration changes, planned maintenance, and uncoordinated customer maintenance that the customer’s noc may not be aware of. If at all possible validate your equipment to see if the outage is on your side. If your not at the location without connectivity call them to ensure they have power. Try to login to the router or PBX that the circuit connects to (on the downed side). If you can login then the your equipment is probably fine. It is very rare for a CSU (the module or port connecting to the carrier) to fail. Tip: if you install a phone line connected to a modem and attach it to the console you can usually get in during most outages which greatly forwards self-service network support. Be sure to have this put in the ticket: if your have validated power and the results, if you can login to the router or PBX, and any other information you think might be useful for their network support staff. Remember, they have to figure out what’s broken and don’t have access to your side without sending someone out. Try to start these checks immediately after you open the ticket. If you can’t validate power or can’t get past the login prompt tell the vendor this, they can still proceed without all the information and may be easier to deal with. Most carriers bill if they dispatch a technician to the customers site and it is an issue with the customer’s equipment or power, but are willing to do anything short of that free of charge. Any dispatch on an issue of the provider’s network is free of charge including previous dispatches if multiple dispatches were made during network support of one issue.
When a circuit is down it is common for the LEC (and only the LEC) to dispatch someone out to the customer premises after other options of network support have failed. Even if it is their side of the network that’s failed they may need someone at the customer’s site to connect to their circuit to run diagnostics. Most will not send a technician out until they have (valid) local contact information (lcon). This is simply the name and phone number(s) of the person who will let the technician in the building and into the equipment room. Try to validate the phone numbers before you give them out (put them on hold or three way if you have to.) Also be sure to include an alternate if at all possible, and be sure if it crosses shift change or goes past business hours it is still valid. If you either think dispatch is a possibility or it has been a couple hours try to do this proactively and give it to the provider before they have to ask. When the technician is dispatched it is common for him to go to the central office (CO) which is the location of the communications equipment on there side. Don’t panic if they don’t show up, although a phone call to confirm dispatch and the stage of network support is fine. If a technician arrives and it is discovered to be an issue with the LEC’s network there is not normally a fee, there is normally a threat of a bill if it is discovered to be an issue on the customer’s side. It is normally acceptable to insist the LEC dispatch to prove their network is fine as part of their network support.
Many circuit issues are intermittent and appear to be fixed when they aren’t. If your circuit is up and the carrier’s network support staff asks to close the ticket ask that it be set to auto-close rather than closed if they will let you. Auto-close means that if the ticket is not contested in the set time period the ticket will be automatically closed. The advantage is that if it is discovered to be an intermittent issue or that the circuit is taking errors it can be contested and the same ticket can be used rather than a new one. When tickets are closed or set for automatic-closure try to get a reason for outage (rfo). Cases that do not have a reason for outage are often an intermittent issue that has not been resolved despite recovered service. Also beware of errors on the circuit which is when some of the data traveling across the circuit is corrupted. Most routers and some PBXimagess collect statistics on the errors. The circuit was probably intrusively tested during network support which generates a very large number of errors. Take this opportunity to clear these counters (”clear counters” on Cisco routers). Check these counters just after they have been cleared and again hours latter (or the next day). A “small” number of errors is tolerable but otherwise, if the errors are incrementing, contest the ticket and have the issue fixed. If there are many errors it isn’t resolved yet. Strictly speaking there shouldn’t be any errors on T1/T3/Frame/ATM circuits. In the ticket state that your seeing errors if this is the case and beware of future intermittent outages and dropped connections.
There are many things that are worth knowing about network support. There isn’t always many options for unsatisfactory handling of the outage. Circuit outages or a degraded circuit can be very expensive for the business if it prevents people from working or from serving the customer. By the way, it is very common for a circuit outage to create an outage on the backup circuit at the same time due to the nature of networking issues. If this happened to you, or you want to prevent it, please read our redundancy guide. Network support can be painful and certainly quite a bit more complex than simply telling the carrier to fix it. Hopefully this guide helped you or at least you learned something useful for next time.