TwistedSNMP
TwistedSNMP is a set of SNMP protocol implementations for Python's
Twisted Matrix networking framework using the PySNMP project. It
provides the following:
- get, set, getnext and getbulk Manager-side queries
- get, set, getnext and getbulk Agent-side services
Eventual goals of the system:
- provide access to all v1 and v2 SNMP functionality for writing
Agent and Manager services
- provide convenient testing mechanisms for SNMP Agent/Manager
development (e.g. mirroring an SNMP Agent's OID tree for local query
testing)
You can download
TwistedSNMP from the project page.
If you have feedback, feel free to contact the author.
Installation
TwistedSNMP is distributed as a Python distutils package. You
can install it by unpacking the source distribution and running:
setup.py install
In the top level of the install directory. You will need both Twisted and PySNMP 3.x or PySNMP-SE
installed.
The PySNMP-SE package (available from the TwistedSNMP
download page) allows for faster scanning, but returns tuple-based
OIDs. PySNMP-SE is recommended for use for all new TwistedSNMP
development.
Documentation
At the moment, most of the documentation for the module is in the
form of the auto-generated PyDoc
reference API, which includes auto-generated documentation for PySNMP. You can find usage examples
in the test sub-directory of the distribution.
Usage
TwistedSNMP is designed to provide natural APIs for dealing with
remote SNMP Agents under the assumption that the code will be used to
implement Manager-side applications. At the moment, it does not
support Trap handling.
For manager-side operations, we can use the port function in the snmpprotocol module to
create a new Twisted port object with an attached SNMPProtocol
object. With this, we can create any number of AgentProxy objects which
provide the API we use for querying the SNMP Agent.
port = snmpprotocol.port()
proxy = agentproxy.AgentProxy(
ipAddress, 161,
community = sys.argv[2],
snmpVersion = 'v2',
protocol = port.protocol,
allowCache = True,
)
Once we have the AgentProxy, we have three major entry-points.
The first is "get", which retrieves single oid values. You can
see a full usage example in simpleget.py:
df = proxy.get(
oids, timeout=.25, retryCount=5
)
the deferred object returned from the get method will fire with a
dictionary mapping { OID: value }
for each oid in oids. OIDs which were not present on the agent
will return None for their value. If the Agent does not return a
value, or returns an error, then the deferred will fire it's errback
chain with the returned error.
The second entry-point is "getTable", which retrieves all sub-oids
for each oid in the list of oids passed in. You can see a full
usage example in simplegettable.py:
df = proxy.getTable(
oids, timeout=.25, retryCount=5
)
the deferred object returned from the getTable method will fire with
a dictionary mapping { rootOID: { OID : value} }
for each rootOID in oids. rootOIDs which were not present on the
agent will
return None instead of the dictionary of { OID : value}
pairs. If the Agent does not return a value, or
returns an error, then the deferred will fire it's errback chain with
the returned error.
The third entry-point is "set", which takes a dictionary of {
OID : value}
pairs and uploads the values to the Agent.
Usage example in simpleset.py:
df = proxy.set(
oidSet, timeout=.25, retryCount=5
)
At the moment the return value for the deferred object is just the
raw SNMP message returned from the agent. This may be changed to
be something useful at some point.
For Agent-side operations, hopefully the simpleserver.py
sample will get you started on how to set up testing agents (keep in
mind that the goal for these agents in the current codebase is solely
to test the Manager-side implementations, so they are rather
primitive). There are bisect (in-memory)
and BSDDB-based
OIDStores available.
Changes
Version 0.3.12
- Latency fixes to make loading of very large tables less likely to
produce long hangs in applications
- Beginning of the work to provide a PySNMPv4 version of the
library (unfinished, may never actually be finished)
Version 0.3.11
- Optimise the integrateNewRecord method used with pysnmp-se to
reduce the number of total iterations required to process the incoming
records
Version 0.3.10
- Fix bug in the Agent code, was trimming the set of OIDs returned
when a particular OID reached end-of-MIB-view, now fills the rest of
the value-sets with end-of-MIB-view objects, so OID sets are always
indexed by the requested OIDs.
Version 0.3.9
- Allow retrieval of the "tail" of a table by specifying marker
OIDs, that is, retrieve from the item beyond the marker OID to the end
of the table, to allow for querying "new items" from a table given
known previous items (assuming new items are appended to the end of the
table)
Version 0.3.8
- Include missing module from the 0.3.8 release (and actually test
with 3.4.x to be sure it works with that version)
- Release the 3.5 extensions (PySNMP-SE) so that others can play
with them
- Repackage with somewhat cleaner distutils script
Version 0.3.7
- Provide method to cache OID encodings (when this feature is
available in PySNMP (currently only in 3.5 extensions))
Version 0.3.6
- Fix bug/undesirable behaviour relating to the caching mechanism,
was using the RequestId object, instead of the value of that object as
the request ID key
Version 0.3.5
- Provides simple mechanism for caching PySNMP get and getbulk
request objects to optimise queries
- Tightens up the allowed values for "oids" and "roots" parameters
throughout, now must be either a sequence of OIDs (or OID-compatible
values) for get/getTable or a sequence of (OID,value) pairs for set
Version 0.3.0
- Allows for patches to PySNMP 3.4 to use tuple-based OID objects
for representing OIDs instead of strings, if the patch is installed
then TwistedSNMP will produce
pysnmp.ans1.oid.OID
instances for all OID return values, otherwise it will continue to
produce formatted-string OIDs.
- The effect of the patch on performance is dependent primary on
how much tabular retrieval you do.
- Note that if you install the patch, you will almost certainly
need to alter your client code to deal with OID objects rather than
strings.
Version 0.2.15
- Uses psyco, when
available, to bind various classes in PySNMP which have been identified
as causing significant slowdowns.
- Splits out the OID-setting code from the message interpretation
code in Agent so that it's easier to programmatically set OIDs on an
Agent.
Version 0.2.14
- Eliminates potential memory leaks by reworking the __del__ method
on BSDOIDStore objects
- Note: There is another memory leak for which I have a
work-around available, but this leak is in Twisted itself (particularly
the deferred object), so it's not part of the TwistedSNMP release
- Reduce log level for "Unexpected request key" message, as it is
extremely common and so floods application logs
Version 0.2.13
- Adds ability to insert sub-OIDStores in BisectOIDStore objects,
this allows you to compose OIDStores such that a given (shared)
sub-storage is responsible for common application-level values
- Adds ability to insert callable objects, this allows you to
insert functions that return calculated values for a given OID.
Functions have the signature:
def callback( oid, storage ):
returning a value appropriate to the given OID
Version 0.2.12
- Work around what appears to be a memory leak in deferred objects
holding references to objects which also hold references to the deferred
- Fix
for memory leak where tabular retrieval times out, was doing an
incorrect test to determine whether to delete the key, so never deleted
it
- Switch to use of Python's standard logging module for all debug
output
- Note: This will require that you set up a logging handler, e.g.
by calling
logging.basicConfig()
during your
application's initialisation procedure
- Switch to using an internal registration point for PySNMP modules
(to avoid problems when PySNMP 4 comes out with an entirely new
organisation), shouldn't affect client modules save if they want to
create
v2c
objects, if so, importing v2c
from twistedsnmp.pysnmpproto
is the way to go to keep in
synch with the version being used by TwistedSNMP.
- Note: PySNMP 4.0 alpha is not supported yet, I'm waiting for
the API to stabilise before trying to support it (it looks like a lot
of non-trivial changes that will require rewriting large tracts of
TwistedSNMP).
- A bit more documentation including usage examples.
Version 0.2.11
- Allow for passing in maxRepetitions for getBulk requests, this
simply controls the size of the window requested from the Agent, which
should allow you to reduce overhead in cases where you know you only
want a small table
Version 0.2.10
- Rewrite table-retrieval results-handling to be more robust and
simple
Version 0.2.9
- Catch socket errors during message sending and report as Twisted
failures (otherwise the errors simply print exceptions and are ignored,
which can hang client code that expects (rightly) an eventual return
value from something which calls a TwistedSNMP method).
Version 0.2.8
- BSDOIDStore.close method added to actually save data to disk
(useful, that :) )
- Modifications to MirrorAgent to call close on OID store
- Bug-fixes in both OIDStore classes (old BSDDB data-files will be
incompatible as a result)
- Essentially, the OIDs were being stored in too naive an order,
so the oid 0.1.2 would sort after 0.1.12, i.e. using simple string
sorting
- BisectOIDStore now uses a tuple of integers while BSDOIDStore
uses a packed binary string representation
Version 0.2.7
- Refactoring of Agent class to allow for easier construction of
mock objects that can simulate agent dialogues without needing to go
through the SNMP encoding/decoding stages
- Introduction of stand-alone errors module
- Use of errors module in OIDStore classes to indicate failed OID
lookup
Version 0.2.6
- Mostly just documentation changes to reflect new project location
URLs and the like.
Version 0.2.5
- Altered timeout methods to catch exceptions within themselves and
relay to the caller via the registered deferred object
- Documentation enhancements
- Patch to work around Python 2.3.x bug in the BSDDB wrapper class
- Eliminated old algorithms for MassRetriever
- Reduced default MassRetriever iteration delay to 0.005 instead of
0.01 seconds
Version 0.2.4
- Fixed bug with table request processing.
Version 0.2.3
- Rework of mass retriever algorithm (yet again) to be more robust
when dealing with larger AgentProxy sets
- Much simpler mechanism
- Throttles to initiate less than 200 queries/second by default,
but ignores the number in play, which, as it turns out, seems to be
somewhat useless for managing load. This is not ideal, as, at
least on our testing systems, the hardware can readily handle a few
thousand queries/second (it's Twisted's structure that requires the
throttling).
- Internal rework so that timeouts are cancelled ASAP for messages
received
- To prevent timeouts issuing before the system gets around to
calling the callback
- Discovered (as-yet-unfixed) bug in Agent's (as distinct from
AgentProxy) processing of table requests
- Basically it doesn't properly handle tables of different
lengths.
Version 0.2.2
- Bug-fix in utilities/mirroragent.py
- Test module cleanup for cross-platform operation
- Allow for operation when no bsddb module is available, as there
are platforms on which it's not available by default
- Use non-privileged port if the privileged 161 is not bindable
- Added check to twineTables for NULL OID set (passing in such an
OID set makes no sense, and would cause silent, difficult-to-track-down
errors)
Version 0.2.1
- Cleaned up and reworked mass-query mechanism;
- patterned loosely after the table-query mechanism
- fixed at least one logic bug in the batching machinery
- added adaptive heuristics for determining number of queries to
keep in play
- changes the api for retrieval in a backward-incompatible way
- Fixed shallow bug in TableRetriever's handling of responses with
unequal rootOID and (OID,value) set lengths
Version 0.2.0
- Complete refactoring of the client/manager-side API.
Instead of requiring one port per managed agent, is able to manage any
number of agents from a single port. Most of the machinery of the old
protocol object is now part of an AgentProxy class. You *will*
need to modify all client code to work with the new API. Mostly
just a matter of a different initialisation pattern and then using the
AgentProxy object in the same manner as you used the old SNMPProtocol
object.
- Crude mass-query mechanism created. As of yet only tested
against a local server, and doesn't yet have balancing heuristics to
throttle load in response to network response, but does appear to work.
- Dropped the snmpports module. It's no longer necessary with
the new structure.
- Bug was discovered in Python 2.3's BSDDB module which affects the
BSDDB storages, bug report submitted to the Python project, but for
now, will not pass tests under Python 2.3.
Version 0.1.5
- Refactored the test-suite into its own package w/ sub-modules for
the different test-types
- Fixed bug in usage of oidStore API for Agent set-requests
Version 0.1.4
- Utilities:
- mirroragent -- mirrors an agent to local bsddb shelve for
testing (still very rough, just a sketch, really)
- BSDOIDStore -- a new OIDStore implementation for use in utilities
- Major refactoring of the Agent-side mechanisms:
- AgentProtocol -- low-level send and dispatch of messages to the
Agent
- Agent -- SNMP logic for get/getNext/getTable/set implementation
- OIDStore -- interface for storage/retrieval of ordered OID tree
- BisectOIDStore -- the demo OIDStore implementation for testing
purposes using the bisect module
Version 0.1.3
- Agent and Manager-side set protocol implementation
- Minimal sample code for Agent-side set code, doesn't yet do the
error-negotiation stuff
Version 0.1.2
- Test code
- Agent-side protocol sample code extended to include getnext and
getbulk
Version 0.1.1
- Beginning of work on an Agent-side protocol and some sample Agent
code
Version 0.1