RADIUS for Device Management: The Server Is Down, and Nobody Can Log In Anywhere

Centralising who may log in to network devices is the least glamorous project in enterprise networking and the one most likely to end with somebody driving to a site with a console cable. The configuration is short. The part that goes wrong is not the configuration — it is the question nobody asked before applying it, which is what the device should do when it cannot reach the server.

The answer to that question is set by two or three words at the end of a method list, and the difference between getting it right and getting it wrong is the difference between a brief authentication delay and an entire site becoming unmanageable during exactly the network failure that made the server unreachable in the first place.

This article covers what the protocol actually protects and what it leaves in clear, how administrative login is authenticated and how a privilege level is assigned, how to configure the fallback behaviour and test it without risk, what per-command control and accounting can and cannot deliver, and the deployments that look complete and are weaker than they appear. It is written for the lab rather than for the written exam, and sits alongside the rest of the CCIE Enterprise Infrastructure lab certification track.

Authentication, authorization and accounting are three independent decisions with three independent method lists, and the fallback method at the end of each list determines what the device does when the server cannot be reached.

What Does the Protocol Actually Protect?

What is encrypted?

The password attribute, obscured using the shared secret. Everything else — the username, the device's identity, the attributes returned, the privilege level granted — travels in clear. Anyone who can observe the path learns who logged in to what and what they were given, without needing the secret. That is a property of the protocol rather than a misconfiguration, and it decides where the servers can safely be placed and whether the transport needs additional protection.

A Deeper Dive into the Protocol

What the shared secret does and does not do

It is used to obscure the password field and to compute an authenticator that lets each side confirm the other knew the secret. It is not a transport encryption key and it does not protect the rest of the packet. A weak secret weakens both functions, and a secret shared across every device in the estate means compromising one device yields the secret for all of them.

The practical standard is a long random secret, different per device or at least per device group, distributed by the configuration management system rather than typed. Anything else is a secret that will be the same value for a decade and will appear in a text file somebody emailed.

The integrity question

A published attack against the protocol's response authenticator showed that an attacker positioned between a device and a server can, under certain conditions, turn a rejection into an acceptance. The attack targets the integrity of the response rather than the confidentiality of the password.

Two mitigations apply and both are worth implementing. Requiring the message authenticator attribute on every packet closes the specific weakness. Carrying the exchange over a protected transport — a dedicated management network, an encrypted tunnel, or a version of the protocol that runs over a secure transport — closes it and the confidentiality problem at the same time.

Where the servers should sit

On a management network reachable from every device by a path that does not traverse anything untrusted. Where that is not achievable — a branch reached over a third-party circuit, for instance — the exchange should be inside a protected tunnel, for the confidentiality reason as much as the integrity one.

This is a design decision made once and it constrains everything else. A deployment that placed the servers correctly needs no further thought about the protocol's weaknesses; one that did not will keep encountering them.

Why this protocol for device administration at all

The traditional answer for administrative access is the other protocol, which encrypts the whole exchange and separates the three functions properly. The reason organisations use this one anyway is identity consolidation: the same server, the same directory integration and the same policy engine already handle network access for every user and device, and adding administrative access to it avoids running a second system.

That is a legitimate trade and it should be made knowingly. What is given up is per-command authorization, which this protocol does not natively provide, and the confidentiality of everything except the password. What is gained is one identity system instead of two. For many organisations the second outweighs the first; for organisations with a regulatory requirement for command-level control, it does not.

! Server definition, with a per-device secret from the
! configuration management system rather than a typed constant
R1(config)# aaa new-model
!
R1(config)# radius server ISE-1
R1(config-radius-server)# address ipv4 10.200.0.10 auth-port 1812 acct-port 1813
R1(config-radius-server)# key 7 <per-device-secret>
R1(config-radius-server)# automate-tester username probe probe-on
!
R1(config)# radius server ISE-2
R1(config-radius-server)# address ipv4 10.200.0.11 auth-port 1812 acct-port 1813
R1(config-radius-server)# key 7 <per-device-secret>
R1(config-radius-server)# automate-tester username probe probe-on

! Group them, and pin the source so the server's device list matches
R1(config)# aaa group server radius ISE-GROUP
R1(config-sg-radius)# server name ISE-1
R1(config-sg-radius)# server name ISE-2
R1(config-sg-radius)# ip radius source-interface Loopback0
R1(config-sg-radius)# deadtime 15
!
! How quickly a server is declared unusable
R1(config)# radius-server dead-criteria time 10 tries 3

The source address, which is not cosmetic

The server identifies the device by the address the request came from, and its device list holds a secret per address. A device whose requests arrive from whichever interface routing happened to choose will sometimes present an address the server does not recognise, and the request is silently ignored — not rejected, ignored, which reads as a timeout.

Pinning the source to a loopback fixes it permanently and should be done in the same change that defines the servers. The symptom otherwise is intermittent authentication failures that correlate with routing changes and with nothing else, which is a genuinely difficult thing to diagnose from the device side.

How Is Administrative Access Authenticated and Authorized?

What are the two steps?

Authentication establishes that the person is who they claim. Authorization establishes what they get, which for administrative access means a privilege level, returned as an attribute alongside the acceptance. Both are configured as method lists and both lists are applied to the lines separately. A configuration with authentication working and authorization missing produces a user who logs in successfully and lands at the lowest privilege level with almost nothing available.

A Deeper Dive into Login and Privilege

The method list and its order

A list names one or more methods to try in order. The critical semantics: a method that answers ends the evaluation, whether the answer was yes or no. Only a method that fails to answer — because the server is unreachable — causes the next method to be tried.

So a list naming the server group and then the local database does exactly the right thing. A rejection from the server is final, which is what centralised control means. An unreachable server falls through to local, which is what keeps the device manageable. Both behaviours come from the same two words and neither is optional.

! Authentication: server, then local if the server does not answer
R1(config)# aaa authentication login VTY-AUTH group ISE-GROUP local
!
! Authorization: the same, plus a fallback for an authenticated user
R1(config)# aaa authorization exec VTY-AUTHZ group ISE-GROUP local if-authenticated
!
! Console keeps its own list, on the local database only
R1(config)# aaa authentication login CONSOLE-AUTH local
R1(config)# aaa authorization exec CONSOLE-AUTHZ local

How a privilege level is returned

The server returns an attribute with the acceptance and the device applies it. Two forms are in common use: a vendor-specific attribute naming the shell privilege level directly, and a standard attribute whose value the device maps to a level. Either works and the first is more explicit, which makes it easier to read on the server and easier to explain during an audit.

Without such an attribute the user is authenticated and receives the lowest level, which allows almost nothing. That is the correct default and it produces the most common support call of any deployment: "I can log in but there are no commands", which is an authorization problem being reported as an authentication success.

! What the server should return with the acceptance
!   cisco-av-pair = "shell:priv-lvl=15"        full access
!   cisco-av-pair = "shell:priv-lvl=1"         read-only shell
! or the standard attribute:
!   Service-Type = Administrative               maps to level 15
!   Service-Type = NAS-Prompt                   maps to level 1
!
! What the device received, for a live session
R1# show users
R1# show privilege
Current privilege level is 15

Applying the lists to the lines

A list does nothing until it is applied. Administrative sessions take theirs from the virtual terminal lines, and the console takes its own. Applying the server-backed list to the console as well is the single most effective way to lose access to a device permanently, and there is no benefit to it that offsets the risk.

! Virtual terminals: the server-backed lists
R1(config)# line vty 0 15
R1(config-line)# login authentication VTY-AUTH
R1(config-line)# authorization exec VTY-AUTHZ
R1(config-line)# transport input ssh
R1(config-line)# exec-timeout 15 0
!
! Console: local only. This is the escape hatch.
R1(config)# line con 0
R1(config-line)# login authentication CONSOLE-AUTH
R1(config-line)# authorization exec CONSOLE-AUTHZ

The local account that must exist first

Created before anything else, with a strong password, at full privilege. It is the fallback for every method list and it is what the console uses. A deployment that creates the method lists before the account has a window in which the device has no working authentication path at all.

Its password belongs in the organisation's credential store, rotated on a schedule, and known to be correct — because the one occasion it will be used is an incident, and discovering then that it does not work is the scenario the account exists to prevent.

Groups, and mapping them to levels

The policy on the server decides which level each group receives, and two or three levels is usually enough: full access for the network team, read-only for everyone with a legitimate reason to look, and possibly an intermediate level for a service desk that performs a defined set of actions.

The intermediate level is where effort is wasted. Constructing a custom privilege level by moving individual commands between levels is possible, laborious and fragile across software versions. Where genuinely granular control is needed, the other protocol's per-command authorization is the mechanism designed for it, and the honest recommendation is to use that rather than to build something elaborate here.

How Do You Avoid Locking Yourself Out?

What is the procedure?

Local account first, then servers, then method lists, then apply to the virtual terminal lines only. Then open a second session and authenticate through it while the first session remains open and untouched. If the second session fails, the first is still there to undo the change. Only when the second session works does the first get closed, and the console keeps its local-only list permanently regardless.

A Deeper Dive into Not Being Locked Out

The second session rule

Every change to authentication is tested from a new session while the existing one stays open. This is the single habit that prevents the outcome this section is about, it costs nothing, and it is skipped constantly because the change looked simple.

The existing session retains its authorization and is not re-evaluated, so it remains usable even when the new configuration rejects everything. That property is what makes the rule work and it is why closing the first session before testing removes the only remaining way in.

Testing without committing

The device can send a test request to the server group and report the result, without any session being involved. This confirms reachability, the shared secret and the server's policy in one command, and it is the right first check after defining the servers and before writing any method list.

! Test the server before any method list depends on it
R1# test aaa group ISE-GROUP alice <password> new-code
User successfully authenticated

USER ATTRIBUTES

username             0   "alice"
priv-lvl             0   15
!
! Server reachability and dead state
R1# show aaa servers | include RADIUS|State|requests

The fallback method, in both lists

Authentication falls back to the local database. Authorization needs its own fallback and a different one: a user who has already authenticated should be authorized without the server, because refusing to authorize a session that the device itself just authenticated is a failure with no upside.

Omitting the authorization fallback is more common than omitting the authentication one, and its effect is subtler: during a server outage users authenticate against the local account and then cannot obtain a usable shell. The device is reachable and useless, which is arguably worse than being unreachable because it wastes time before anyone reaches for the console.

Dead server detection, and the delay it causes

When the server is unreachable, the device waits for a timeout and retries before declaring it dead and moving to the next method. With default values applied to two servers in sequence, the wait before a local login succeeds can be long enough that the operator assumes the session has failed and gives up.

Tuning the criteria — a short timeout, few retries, and a dead time during which a server declared dead is not retried — reduces that to a few seconds. Combined with a periodic probe that marks a recovered server alive again, this makes the fallback fast enough to be usable during an incident rather than merely correct.

! Make the fallback fast enough to be usable under pressure
R1(config)# radius-server dead-criteria time 5 tries 2
R1(config)# radius-server timeout 3
R1(config)# radius-server retransmit 1
!
R1(config)# aaa group server radius ISE-GROUP
R1(config-sg-radius)# deadtime 10
!
! Which servers are currently considered dead
R1# show aaa servers | include host|State

The console, permanently local

There is no configuration in which putting the console behind the server improves anything. The console is the path used when the network is broken, and the network being broken is precisely the condition in which the server is unreachable. Leaving it on the local database costs nothing and is the difference between a remote fix and a site visit.

Physical access to the console is a separate control and belongs to facilities rather than to the network configuration. Treating the console's local authentication as a security weakness leads to a decision that trades a real operational capability for a marginal theoretical gain.

Verifying the failure mode deliberately

Reading the configuration is not verification. The test is to make the server unreachable — in a lab, or on one device during a window — and confirm that a login using the local account succeeds, reaches full privilege, and does so quickly enough to be usable.

Doing it once, per platform and per software version, is enough. Not doing it means the fallback is a belief rather than a fact, and it will be tested for the first time during an incident. Reading this once is not the same as being able to do it under time pressure, which is what repetition against realistic CCIE lab practice scenarios is for.

What Can Be Controlled and Recorded?

What is available?

Accounting is comprehensive: session start and stop, and a record of commands entered at chosen privilege levels, all sent to the server as they happen. Per-command authorization is not natively available in this protocol — the granularity it offers is the privilege level assigned at login. Where individual commands must be permitted or denied per user, that requires either the other protocol or an external mechanism, and pretending otherwise produces an audit finding rather than a control.

A Deeper Dive into Control and Records

What accounting records

Session records give who connected, from where, when, and for how long. Command records give each command entered at a configured privilege level, with the user and the timestamp. Together they answer the question an incident review asks — who changed this, and when — without depending on anyone's memory.

Command accounting at the highest privilege level is the one most worth having and the cheapest to enable. It records the changes that matter and generates a manageable volume, where recording every command at every level produces a stream that nobody reads.

! Sessions, and commands at the level that matters
R1(config)# aaa accounting exec default start-stop group ISE-GROUP
R1(config)# aaa accounting commands 15 default start-stop group ISE-GROUP
!
! Configuration changes, including who made them
R1(config)# aaa accounting system default start-stop group ISE-GROUP
!
! Is anything actually being sent?
R1# show aaa servers | include Accounting|requests

Accounting must not block login

If the accounting server is unreachable, the device should still permit administrative access. The default behaviour is generally to proceed, and it is worth confirming rather than assuming, because a configuration that blocks a session when its accounting record cannot be delivered converts an accounting outage into an access outage.

The correct posture is that accounting is best effort for administrative access. Records are valuable and are not worth locking the operations team out of the network to guarantee.

Privilege levels, and their limits

The level assigned at login determines what is available. Levels can be customised by moving individual commands into them, which sounds like per-command control and is not: the level is a property of the device, not of the user, so every user given that level gets the same set, and the set must be maintained identically on every device and revalidated after every software upgrade.

That maintenance burden is the reason the approach is usually a mistake. Two or three standard levels, applied consistently, are maintainable. A bespoke level per team is not, and it degrades silently as devices and versions diverge.

! A custom level is a device property, not a user property
R1(config)# privilege exec level 5 show running-config
R1(config)# privilege exec level 5 clear counters
!
! Every device needs this identically, and it must be
! revalidated after every upgrade. This is the cost.
R1# show privilege
R1# show running-config | include privilege

Where genuine command control is required

A regulatory or contractual requirement for per-command authorization is not satisfied by privilege levels, and saying otherwise during an audit is a poor position. The protocol designed for this separates authorization from authentication and evaluates each command against policy, and that is the correct answer.

Running both is entirely workable: this protocol for network access, the other for device administration, against the same identity store and the same policy engine. It is more moving parts and it is honest about what each protocol does. The decision belongs to whoever owns the requirement rather than to the network team alone.

Dynamic authorization

The server can send an instruction to an already-established session — to disconnect it, or to change what it is permitted. For network access this is central to how policy changes are enforced without waiting for a reauthentication.

For administrative sessions its main use is disconnection: an account disabled in the directory can have its live sessions terminated rather than left running until the operator logs out. Enabling it requires the device to accept these instructions from the server and to authenticate them with a secret, and it is worth having for that one capability alone.

! Accept instructions from the server for live sessions
R1(config)# aaa server radius dynamic-author
R1(config-locsvr-da-radius)# client 10.200.0.10 server-key 7 <secret>
R1(config-locsvr-da-radius)# client 10.200.0.11 server-key 7 <secret>
R1(config-locsvr-da-radius)# auth-type all
!
R1# show aaa servers | include Dynamic

Which Deployments Are Weaker Than They Look?

What should be checked?

Five things that a working login does not prove. Whether a local fallback exists in both lists. Whether the console was left local. Whether the source address is pinned so the server recognises every device. Whether the shared secret is unique per device rather than shared across the estate. And whether the failure mode has been tested rather than reasoned about. A deployment can pass every functional test and fail all five.

A Deeper Dive into the Weak Points

The estate-wide shared secret

One secret on every device is convenient to deploy and means that reading the configuration of any device — a decommissioned switch sold for parts, a backup file on a badly protected share, a device an attacker reached — yields the secret protecting authentication for every device in the organisation.

Per-device secrets are only practical with configuration management generating them, which most organisations have. Where they do not, a secret per site or per device class is a substantial improvement over one secret everywhere and is achievable manually.

Configuration backups, which contain everything

The secret appears in the configuration, obscured by a reversible encoding that is not encryption. Every backup of every device contains it in a form anyone can recover. This is not a flaw in the protocol; it is a property of where the secret has to live.

The consequence is that the backup repository deserves the same protection as the devices. Treating backups as ordinary files, on an ordinary share, with ordinary access, undoes the protection the secret provides.

The device that was never onboarded

Estates always contain devices the project missed — a switch in a cupboard, a router in a lab that quietly became production, a device commissioned during the project and configured from an old template. Each one still has a shared local account with a password that has not changed since it was built.

Finding them is a reconciliation exercise: the server's device list against the inventory against what actually answers on the management network. All three will differ and the differences are the finding. This is the most valuable single check available after a deployment and it is almost never done.

! The five checks, per device, scriptable
R1# show running-config | include aaa authentication login
R1# show running-config | include aaa authorization exec
R1# show running-config | section line con 0
R1# show running-config | include ip radius source-interface
R1# show aaa servers | include host|State
!
! And the local accounts that still exist
R1# show running-config | include ^username

Local accounts nobody removed

After centralisation, the per-person local accounts that preceded it usually remain. They work, they bypass the central system entirely, and they are invisible in the server's records because the server never sees those logins.

One local account should remain: the break-glass account, documented and rotated. Everything else should be removed, and the removal verified per device rather than assumed from a template. An account left on forty devices out of four hundred is exactly the kind of gap that is found by an auditor rather than by the team.

Testing the failure mode, once

The fallback path is the part of the design most likely to be wrong and least likely to be exercised. Testing it means making the servers unreachable and confirming that a local login succeeds at full privilege within a few seconds, on each platform and software version in the estate.

Once is enough, and once is more than most deployments manage. Without it the fallback is an assumption, and an assumption about what happens during an outage is the least useful kind.

What to record for the next engineer

Where the break-glass credentials live and how often they are rotated. Which method lists are applied to which lines and why the console differs. The tuned dead-detection values and the measured fallback delay. And the date the failure mode was last tested.

That last item is the one that decays. A fallback tested three years and two software versions ago is not evidence of anything, and recording the date is what makes its staleness visible.

Blueprint framing

The CCIE Enterprise Infrastructure v1.1 blueprint covers device access control and AAA within its infrastructure security and services domain. What is asked is generally the method list semantics — what happens on a rejection as opposed to a timeout — and the distinction between authentication, authorization and accounting, rather than a full server deployment.

Conclusion

The protocol protects the password and nothing else. Usernames, device identities and the attributes returned all travel in clear, which makes where the servers sit and how the path is protected a design decision that configuration cannot fix afterwards. Using this protocol rather than the one designed for administrative access is a legitimate trade — one identity system instead of two — and what is given up is per-command authorization, which privilege levels do not replace however carefully they are constructed.

The three functions are independent and each has its own method list. Authentication establishes identity, authorization returns a privilege level, and a configuration with the first and not the second produces users who log in successfully and find nothing available. Both lists need a fallback and they need different ones: the local database for authentication, and authorization for anyone the device itself has already authenticated.

What decides whether the deployment is sound is the behaviour when the server cannot be reached, because that condition arrives during exactly the network failure somebody is trying to diagnose. A rejection from the server is an answer and ends the evaluation, which is what centralised control means; an unreachable server is not an answer and falls through, which is what keeps the site manageable. Leave the console local, pin the source address, test the fallback by actually breaking the path rather than by reading the configuration, and reconcile the server's device list against the real inventory — because the devices the project missed still have the password they were built with. More CCIE Enterprise Infrastructure material — labs, protocol breakdowns and study guides — is collected on the SPOTO CCIE site.

External Links

• RFC 2865 — Remote Authentication Dial In User Service (RADIUS)
• RFC 2866 — RADIUS Accounting
• RFC 5176 — Dynamic Authorization Extensions to RADIUS
• RFC 6614 — Transport Layer Security Encryption for RADIUS
• RFC 8907 — The TACACS+ Protocol
• RFC 3579 — RADIUS Support for EAP
• Cisco Learning Network — CCIE Enterprise Infrastructure

Reference Notes

1. RFC 2865 specifies RADIUS, in which the User-Password attribute is hidden using the shared secret while other attributes, including User-Name, are carried without confidentiality.
2. RFC 2865 assigns UDP port 1812 for RADIUS authentication, and notes the earlier use of port 1645.
3. RFC 2865 defines the Service-Type attribute, whose Administrative and NAS-Prompt values are commonly mapped by network devices to administrative and read-only privilege.
4. RFC 2865 defines the NAS-IP-Address attribute identifying the requesting device, which is why the source address used for requests must match the server's device entry.
5. RFC 2866 specifies RADIUS accounting, including start and stop records, carried on UDP port 1813 with 1646 as the earlier assignment.
6. RFC 3579 defines the Message-Authenticator attribute, which provides integrity protection for RADIUS packets and mitigates attacks against the response authenticator.
7. RFC 5176 specifies dynamic authorization extensions, including Disconnect and Change-of-Authorization messages sent by the server to an active session.
8. RFC 6614 specifies RADIUS over TLS, which provides confidentiality and integrity for the entire exchange rather than for the password alone.
9. RFC 8907 documents the TACACS+ protocol, which runs over TCP, obfuscates the entire packet body, and separates authentication, authorization and accounting so that individual commands may be authorized.
10. Cisco documentation describes AAA method lists, in which methods are tried in order and a method that returns a response — including a rejection — ends the evaluation, while an unresponsive method causes the next to be tried.
11. Cisco documentation describes the cisco-av-pair vendor specific attribute shell:priv-lvl, used to assign an EXEC privilege level to an authenticated administrative session.
12. The CCIE Enterprise Infrastructure v1.1 unified exam topics include device access control and AAA within the infrastructure security and services domain.