pysmi - SNMP SMI compiler
The PySMI library and tools are designed to parse, verify and transform SNMP SMI MIB modules from their original ASN.1 form into JSON or pysnmp representation.
PySMI library is highly modular. The top-level component is called compiler and it acts as main user-facing object. Most of other components are plugged into the compiler object prior to its use.
Normally, user asks compiler to perform certain transformation of named MIB module. Compiler will:
Search its data sources for given MIB module (identified by name) noting their last modification times.
Search compiler-managed repositories of already converted MIB modules for modules that are more recent than corresponding source MIB module.
If freshly transformed MIB module is found, processing stops here.
Otherwise compiler passes ASN.1 MIB module content to the lexer component.
Lexer returns a sequence of tokenized ASN.1 MIB contents. Compiler then passes that sequence of tokens to the parser component.
Parser runs LR algorithm on tokenized MIB thus transforming MIB contents into Abstract Syntax Tree (AST) and also noting what other MIB modules are referred to from the MIB being parsed.
In case of parser failure, what is usually an indication of broken ASN.1 MIB syntax, compiler may attempt to fetch pre-transformed MIB contents from configured source. This process is called borrowing in PySMI.
In case of successful parser completion, compiler will pass produced AST to code generator component.
Code generator walks its input AST and performs actual data transformation.
The above steps may be repeated for each of the MIB modules referred to as parser figures out. Once no more unresolved dependencies remain, compiler will call its writer component to store all transformed MIB modules.
The location of ASN.1 MIB modules and flavor of their syntax, as well as desired transformation format, is determined by respective components chosen and configured to compiler.
The mibdump.py tool is a command-line frontend to the PySMI library. This tool can be used for automatic downloading and transforming SNMP MIB modules into various formats.
$ mibdump.py --help Synopsis: SNMP SMI/MIB files conversion tool Documentation: http://snmplabs.com/pysmi Usage: mibdump.py [--help] [--version] [--quiet] [--debug=<all|borrower|codegen|compiler|grammar|lexer|parser|reader|searcher|writer>] [--mib-source=<URI>] [--mib-searcher=<PATH|PACKAGE>] [--mib-stub=<MIB-NAME>] [--mib-borrower=<PATH>] [--destination-format=<FORMAT>] [--destination-directory=<DIRECTORY>] [--cache-directory=<DIRECTORY>] [--disable-fuzzy-source] [--no-dependencies] [--no-python-compile] [--python-optimization-level] [--ignore-errors] [--build-index] [--rebuild] [--dry-run] [--no-mib-writes] [--generate-mib-texts] [--keep-texts-layout] <MIB-NAME> [MIB-NAME [...]]] Where: URI - file, zip, http, https, ftp, sftp schemes are supported. Use @mib@ placeholder token in URI to refer directly to the required MIB module when source does not support directory listing (e.g. HTTP). FORMAT - pysnmp, json, null
When JSON destination format is requested, for each MIB module mibdump.py will produce a JSON document containing all MIB objects. For example, IF-MIB module in JSON form would look like:
{ "ifMIB": { "name": "ifMIB", "oid": "1.3.6.1.2.1.31", "class": "moduleidentity", "revisions": [ "2007-02-15 00:00", "1996-02-28 21:55", "1993-11-08 21:55" ] }, ... "ifTestTable": { "name": "ifTestTable", "oid": "1.3.6.1.2.1.31.1.3", "class": "objecttype", "maxaccess": "not-accessible" }, "ifTestEntry": { "name": "ifTestEntry", "oid": "1.3.6.1.2.1.31.1.3.1", "class": "objecttype", "maxaccess": "not-accessible", "augmention": { "name": "ifTestEntry", "module": "IF-MIB", "object": "ifEntry" } }, "ifTestId": { "name": "ifTestId", "oid": "1.3.6.1.2.1.31.1.3.1.1", "class": "objecttype", "syntax": { "type": "TestAndIncr", "class": "type" }, "maxaccess": "read-write" }, ... }
In general, JSON MIB captures all aspects of original (ASN.1) MIB contents and layout. The snippet above is just an example, here is the complete IF-MIB.json file.
The --mib-source option can be given multiple times. Each instance of --mib-source must specify a URL where ASN.1 MIB modules should be looked up and downloaded from. At this moment three MIB sourcing methods are supported:
Local files. This could be a top-level directory where MIB files are located. Subdirectories will be automatically traversed as well. Example: file:///usr/share/snmp
ZIP archives containing MIB files. Subdirectories and embedded ZIP archives will be automatically traversed. Example: zip://mymibs.zip
HTTP/HTTPS. A fully specified URL where MIB module name is specified by a @mib@ placeholder. When specific MIB is looked up, PySMI will replace that placeholder with MIB module name it is looking for. Example: http://mibs.snmplabs.com/asn1/@mib@
SFTP/FTP. A fully specified URL including FTP username and password. MIB module name is specified by a @mib@ placeholder. When specific MIB is looked up, PySMI will replace that placeholder with MIB module name it is looking for. Example: http://mibs.snmplabs.com/asn1/@mib@
When trying to fetch a MIB module, the mibdump.py tool will try each of configured --mib-source transports in order of specification till first successful hit.
By default mibdump.py will search:
file:///usr/share/snmp
http://mibs.snmplabs.com/asn1/@mib@
Once another --mib-source option is given, those defaults will not be used and should be manually given to mibdump.py if needed.
There is no single convention on how MIB module files should be named. By default mibdump.py will try a handful of guesses when trying to find a file containing specific MIB module. It will try upper and lower cases, a file named after MIB module, try adding different extensions to a file (.mib, .my etc), try adding/cutting the '-MIB' part of the file name. If nothing matches, mibdump.py will consider that probed --mib-source does not contain MIB module it is looking for.
There is a small chance, though, that fuzzy natching may result in getting a wrong MIB. If that happens, you can disable the above fuzzyness by giving mibdump.py the --disable-fuzzy-source flag.
It well may happen that many MIB modules refer to a common single MIB module. In that case mibdump.py may transform it many times unless you tell mibdump.py where to search for already transformed MIBs. That place could of course be a directory where mibdump.py writes its transforms into and/or some other local locations.
The --mib-searcher option specifies either local directory or importable Python package (applicable to pysnmp transformation) containing transformed MIB modules. Multiple --mib-searcher options could be given, mibdump.py will use each of them in order of specification till first hit.
If no transformed MIB module is found, mibdump.py will go on running its full transformation cycle.
By default mibdump.py will use:
--mib-searcher=$HOME/.pysnmp/mibs
--mib-searcher=pysnmp_mibs
Once another --mib-searcher option is given, those defaults will not be used and should be manually given to mibdump.py if needed.
Some MIBs may not be automatically transformed into another form and therefore must be explicitly excluded from processing. Such MIBs are normally manually implemented for each target MIB format. Examples include MIBs containing base SMI types or ASN.1 MACRO definitions (SNMPv2-SMI, SNMPV2-TC), initially compiled but later manually modified MIBs and others.
Default list of blacklisted MIBs for pysnmp transformation target is: RFC-1212, RFC-1215, RFC1065-SMI, RFC1155-SMI, RFC1158-MIB, RFC1213-MIB, SNMP-FRAMEWORK-MIB, SNMP-TARGET-MIB, SNMPv2-CONF, SNMPv2-SMI, SNMPv2-TC, SNMPv2-TM, TRANSPORT-ADDRESS-MIB.
If you need to modify this list use the --mib-stub option.
Curiously enough, some MIBs coming from quite prominent vendors appear syntactically incorrect. That leads to MIB compilers fail on such MIBs. While many MIB compiler implementations (PySMI included) introduce workarounds and grammar relaxations allowing slightly broken MIBs to compile, however severely broken MIBs can't be reliably compiled.
As another workaround PySMI offers the borrow feature. It allows PySMI to fetch already transformed MIBs even if corresponding ASN.1 MIB can't be found or parsed.
Default source of pre-compiled MIBs for pysnmp target is:
http://mibs.snmplabs.com/pysnmp/fulltexts/@mib@
http://mibs.snmplabs.com/pysnmp/notexts/@mib@
If you wish to modify this default list use one or more --mib-borrower options.
PySMI design allows many transformation formats to be supported in form of specialized code generation components. At the moment PySMI can produce MIBs in form of pysnmp classes and JSON documents.
JSON document schema is chosen to preserve as much of MIB information as possible. There's no established JSON schema known to the authors.
By default mibdump.py writes pysnmp MIBs into:
$HOME/.pysnmp/mibs (on UNIX)
@HOME@PySNMP ConfigurationMIBs (on Windows)
and JSON files in current working directory.
Use --destination-directory option to change default output directory.
By default PySMI will avoid creating new transformations if fresh enough versions already exist. By using --rebuild option you could trick PySMI doing requested transformation for all given MIB modules.
By default PySMI will stop on first fatal error occurred during transformations of a series of MIBs. If you wish PySMI to ignore fatal errors and therefore skipping failed MIB, use the --ignore-errors option.
Keep in mind that skipping transformation of MIBs that are imported by other MIBs might make dependant MIBs inconsistent for use.
Most MIBs rely on other MIBs for their operations. This is indicated by the IMPORT statement in ASN.1 language. PySMI attempts to transform all MIBs IMPORT'ed by MIB being transformed. That is done in recursive manner.
By using --no-dependencies flag you can tell PySMI not to transform any MIBs other than those explicitly requested to be transformed.
Keep in mind that skipping dependencies may make the whole set of transformed MIBs inconsistent.
Most MIBs are very verbose. They contain many human-oriented descriptions and clarifications written in plain English. Those texts may be useful for MIB browser applications (to display those texts to human operator) but might not make any sense in other applications.
To save space and CPU time, PySMI does not by default include those texts into transformed MIBs. However this can be reverted by adding --generate-mib-texts option.
When MIB texts are generated, whitespaces and new lines are stripped by default. Sometimes that breaks down ASCII art should it occur in MIB texts. To preserve original text formatting, --keep-texts-layout option may be used.
If --build-index option is given, depending on the destination format chosen, the mibdump.py tool may create new (or update existing) document containing MIB information in a form that is convenient for querying cornerstone properties of MIB files.
For example, building JSON index for IP-MIB.json, TCP-MIB.json and UDP-MIB.json MIB modules would emit something like this:
{ "compliance": { "1.3.6.1.2.1.48.2.1.1": [ "IP-MIB" ], "1.3.6.1.2.1.49.2.1.1": [ "TCP-MIB" ], "1.3.6.1.2.1.50.2.1.1": [ "UDP-MIB" ] }, "identity": { "1.3.6.1.2.1.48": [ "IP-MIB" ], "1.3.6.1.2.1.49": [ "TCP-MIB" ], "1.3.6.1.2.1.50": [ "UDP-MIB" ] }, "oids": { "1.3.6.1.2.1.4": [ "IP-MIB" ], "1.3.6.1.2.1.5": [ "IP-MIB" ], "1.3.6.1.2.1.6": [ "TCP-MIB" ], "1.3.6.1.2.1.7": [ "UDP-MIB" ], "1.3.6.1.2.1.49": [ "TCP-MIB" ], "1.3.6.1.2.1.50": [ "UDP-MIB" ] } }
With this example, compliance and identity keys point to MODULE-COMPLIANCE and MODULE-IDENTITY MIB objects, oids list top-level OIDs branches defined in MIB modules. Full index build over thousands of MIBs could be seen here.
There are a few options that may improve PySMI performance.
The --cache-directory option may be used to point to a temporary writable directory where PySMI parser (e.g. Ply) would store its lookup tables.
By default PySMI performing transformation into pysnmp format will also pre-compile Python source into interpreter bytecode. That takes some time and space. If you wish not to cache Python bytecode or to do that later, use the --no-python-compile option.
The mibcopy.py tool attempts to normalize the file name of the MIB file.
It turned out that sometimes vendors name their MIBs in any possible way, not necessarily after the canonical MIB name. This causes problems to the MIB consumers as they may not be able to locate the MIB they need on the file system.
The way how mibcopy.py works is that it tries to read the MIB from the given file (or all files from a given directory or archive), parse MIB's canonical name from the contents of the file. Based on that, the tool tries to rename MIB file into the name which is the same as canonical MIB name. If mibcopy.py encounters the same named file already present on the file system, it reads it up to see its revision date. Then the tool compares the revision dates of the colliding MIB files and either overrides the offending file or drops the file being copied as outdated.
The ultimate goal is to end up with the latest versions of the MIB files all named after their canonical names.
$ mibcopy.py --help Synopsis: SNMP SMI/MIB files copying tool. When given MIB file(s) or directory(ies) on input and a destination directory, the tool parses MIBs to figure out their canonical MIB module name and the latest revision date, then copies MIB module on input into the destination directory under its MIB module name *if* there is no such file already or its revision date is older. Documentation: http://snmplabs.com/pysmi Usage: mibcopy.py [--help] [--version] [--verbose] [--quiet] [--debug=<all|borrower|codegen|compiler|grammar|lexer| parser|reader|searcher|writer>] [--mib-source=<URI>] [--cache-directory=<DIRECTORY>] [--ignore-errors] [--dry-run] <SOURCE [SOURCE...]> <DESTINATION> Where: URI - file, zip, http, https, ftp, sftp schemes are supported. Use @mib@ placeholder token in URI to refer directly to the required MIB module when source does not support directory listing (e.g. HTTP).
The --mib-source option can be given multiple times. Each instance of --mib-source must specify a URL where ASN.1 MIB modules should be looked up and downloaded from. At this moment three MIB sourcing methods are supported:
Local files. This could be a top-level directory where MIB files are located. Subdirectories will be automatically traversed as well. Example: file:///usr/share/snmp
ZIP archives containing MIB files. Subdirectories and embedded ZIP archives will be automatically traversed. Example: zip://mymibs.zip
HTTP/HTTPS. A fully specified URL where MIB module name is specified by a @mib@ placeholder. When specific MIB is looked up, PySMI will replace that placeholder with MIB module name it is looking for. Example: http://mibs.snmplabs.com/asn1/@mib@
SFTP/FTP. A fully specified URL including FTP username and password. MIB module name is specified by a @mib@ placeholder. When specific MIB is looked up, PySMI will replace that placeholder with MIB module name it is looking for. Example: http://mibs.snmplabs.com/asn1/@mib@
When trying to fetch a MIB module, the mibcopy.py tool will try each of configured --mib-source transports in order of specification till first successful hit.
By default mibcopy.py will search:
file:///usr/share/snmp
http://mibs.snmplabs.com/asn1/@mib@
Once another --mib-source option is given, those defaults will not be used and should be manually given to mibcopy.py if needed.
The mibcopy.py writes MIBs into the <DESTINATION> directory.
By default PySMI will stop on first fatal error occurred during transformations of a series of MIBs. If you wish PySMI to ignore fatal errors and therefore skipping failed MIB, use the --ignore-errors option.
Keep in mind that skipping transformation of MIBs that are imported by other MIBs might make dependant MIBs inconsistent for use.
The --cache-directory option may be used to point to a temporary writable directory where PySMI parser (e.g. Ply) would store its lookup tables. That should improve PySMI performance a tad bit.
The MibCompiler object is the top-most interface to PySMI library features. It holds together the otherwise isolated pieces of the compiler infrastructure and manages the workflow of ASN.1 MIB transformation.
This example showcases some of its features:
from pysmi.reader import HttpReader from pysmi.searcher import StubSearcher from pysmi.writer import CallbackWriter from pysmi.parser import SmiStarParser from pysmi.codegen import JsonCodeGen from pysmi.compiler import MibCompiler inputMibs = ['IF-MIB', 'IP-MIB'] httpSources = [('mibs.snmplabs.com', 80, '/asn1/@mib@')] # store compiled MIBs by calling this function def store_mibs(mibName, jsonDoc, cbCtx): print('# MIB module %s' % mibName) print(jsonDoc) mibCompiler = MibCompiler( SmiStarParser(), JsonCodeGen(), CallbackWriter(store_mibs) ) # pull ASN.1 MIBs over HTTP mibCompiler.addSources(*[HttpReader(*x) for x in httpSources]) # never recompile MIBs with ASN.1 MACROs mibCompiler.addSearchers(StubSearcher(*JsonCodeGen.baseMibs)) status = mibCompiler.compile(*inputMibs) print(status)
- class pysmi.compiler.MibCompiler(parser, codegen, writer)
Top-level, user-facing, composite MIB compiler object.
MibCompiler implements high-level MIB transformation processing logic. It executes its actions by calling the following specialized objects:
readers - to acquire ASN.1 MIB data
searchers - to see if transformed MIB already exists and no processing is necessary
parser - to parse ASN.1 MIB into AST
code generator - to perform actual MIB transformation
borrowers - to fetch pre-transformed MIB if transformation is impossible
writer - to store transformed MIB data
Required components must be passed to MibCompiler on instantiation. Those components are: parser, codegenerator and writer.
Optional components could be set or modified at later phases of MibCompiler life. Unlike singular, required components, optional one can be present in sequences to address many possible sources of data. They are readers, searchers and borrowers.
Creates an instance of MibCompiler class.
- Parameters
parser -- ASN.1 MIB parser object
codegen -- MIB transformation object
writer -- transformed MIB storing object
- addBorrowers(*borrowers)
Add more transformed MIBs repositories to borrow MIBs from.
Whenever MibCompiler.compile encounters MIB module which neither of the searchers can find or fetched ASN.1 MIB module can not be parsed (due to syntax errors), these borrowers objects will be invoked in order of their addition asking each if already transformed MIB can be fetched (borrowed).
- Parameters
borrowers -- borrower object(s)
- Returns
reference to itself (can be used for call chaining)
- addSearchers(*searchers)
Add more transformed MIBs repositories.
MibCompiler.compile will invoke each of configured searcher objects in order of their addition asking each if already transformed MIB module already exists and is more recent than specified.
- Parameters
searchers -- searcher object(s)
- Returns
reference to itself (can be used for call chaining)
- addSources(*sources)
Add more ASN.1 MIB source repositories.
MibCompiler.compile will invoke each of configured source objects in order of their addition asking each to fetch MIB module specified by name.
- Parameters
sources -- reader object(s)
- Returns
reference to itself (can be used for call chaining)
- compile(*mibnames, **options)
Transform requested and possibly referred MIBs.
The compile method should be invoked when MibCompiler object is operational meaning at least sources are specified.
Once called with a MIB module name, compile will:
fetch ASN.1 MIB module with given name by calling sources
make sure no such transformed MIB already exists (with searchers)
parse ASN.1 MIB text with parser
perform actual MIB transformation into target format with code generator
may attempt to borrow pre-transformed MIB through borrowers
write transformed MIB through writer
The above sequence will be performed for each MIB name given in mibnames and may be performed for all MIBs referred to from MIBs being processed.
- Parameters
mibnames -- list of ASN.1 MIBs names
options -- options that affect the way PySMI components work
- Returns
A dictionary of MIB module names processed (keys) and MibStatus class instances (values)
MibStatus class instance is used by MibCompiler.compiler() to indicate the outcome of MIB transformation operation.
- class pysmi.compiler.MibStatus
Indicate MIB transformation result.
MibStatus is a subclass of Python string type. Some additional attributes may be set to indicate the details.
The following MibStatus class instances are defined:
compiled - MIB is successfully transformed
untouched - fresh transformed version of this MIB already exisits
failed - MIB transformation failed. error attribute carries details.
unprocessed - MIB transformation required but waived for some reason
missing - ASN.1 MIB source can't be found
borrowed - MIB transformation failed but pre-transformed version was used
PySMI offers a handful of distinct transport mechanisms for fetching MIBs by name from specific locations. In all cases MIB module name to file name match may not be exact -- some name fuzzying can be performed to mitigate possible changes to MIB file name.
FileReader class instance looks up MIB files in given directories on the host running PySMI.
- class pysmi.reader.localfile.FileReader(path, recursive=True, ignoreErrors=True)
Fetch ASN.1 MIB text by name from local file.
FileReader class instance tries to locate ASN.1 MIB files by name, fetch and return their contents to caller.
Create an instance of FileReader serving a directory.
- Parameters
path (str) -- directory to search MIB files
- Keyword Arguments
recursive (bool) -- whether to include subdirectories
ignoreErrors (bool) -- ignore filesystem access errors
ZipReader class instance looks up MIB files in local ZIP archive. ZIP subdirectories and embedded ZIP archives would be traversed.
- class pysmi.reader.zipreader.ZipReader(path, ignoreErrors=True)
Fetch ASN.1 MIB text by name from a ZIP archive.
ZipReader class instance tries to locate ASN.1 MIB files by name, fetch and return their contents to caller.
Create an instance of ZipReader serving a ZIP archive.
- Parameters
path (str) -- path to ZIP archive containing MIB files
- Keyword Arguments
ignoreErrors (bool) -- ignore ZIP archive access errors
HttpReader class instance tries to download MIB files using configured URL.
- class pysmi.reader.httpclient.HttpReader(host, port, locationTemplate, timeout=5, ssl=False)
Fetch ASN.1 MIB text by name from a web site.
HttpReader class instance tries to download ASN.1 MIB files by name and return their contents to caller.
Create an instance of HttpReader bound to specific URL.
NOTE:
The http_proxy and https_proxy environment variables are respected by the underlying urllib stdlib module.
- Parameters
host (str) -- domain name or IP address of web server
port (int) -- TCP port web server is listening
locationTemplate (str) -- location part of the URL optionally containing @mib@ magic placeholder to be replaced with MIB name. If @mib@ magic is not present, MIB name is appended to locationTemplate
- Keyword Arguments
timeout (int) -- response timeout
ssl (bool) -- access HTTPS web site
FtpReader class instance tries to download MIB files from configured FTP server.
- class pysmi.reader.ftpclient.FtpReader(host, locationTemplate, timeout=5, ssl=False, port=21, user='anonymous', password='anonymous@')
Fetch ASN.1 MIB text by name from FTP server. FtpReader class instance tries to download ASN.1 MIB files by name and return their contents to caller.
Create an instance of FtpReader bound to specific FTP server directory.
- Parameters
host (str) -- domain name or IP address of web server
locationTemplate (str) -- location part of the directory containing @mib@ magic placeholder to be replaced with MIB name fetch.
- Keyword Arguments
timeout (int) -- response timeout
ssl (bool) -- access HTTPS web site
port (int) -- TCP port web server is listening
user (str) -- username at FTP server
password (str) -- password for username at FTP server
CallbackReader class instance tries to fetch MIB files by calling user object.
- class pysmi.reader.callback.CallbackReader(cbFun, cbCtx=None)
Fetch ASN.1 MIB text by name by calling user-defined callable.
CallbackReader class instance tries to retrieve ASN.1 MIB files by name and return their contents to caller.
Create an instance of CallbackReader bound to specific URL.
- Parameters
cbFun (callable) -- user callable accepting MIB name and cbCtx objects
- Keyword Arguments
cbCtx (object) -- user object that can be used to communicate state information between user-scope code and the cbFun callable scope
There are cases when MIB transformation may or must not be performed. Such cases include:
foundation MIBs containing manually implemented pieces or ASN.1 MACRO's
obsolete MIBs fully reimplemented within modern MIBs
already transformed MIBs
MibCompiler expects user to supply a searcher object that would allow or skip MIB transformation for particular name based on whatever reason it is aware of.
In general, searcher logic is specific to target format. At the time being, only pysnmp code generation backend requires such filtering.
Transformed MIBs that were saved in form of Python files can be checked with PyFileSearcher class instances.
- class pysmi.searcher.pyfile.PyFileSearcher(path)
Figures out if given Python file (source or bytecode) exists at given location.
Create an instance of PyFileSearcher bound to specific directory.
- Parameters
path (str) -- path to local directory
Some MIBs, most frequently the base ones, can be stored at a Python package. There existence can be checked with the PyPackageSearcher class instances.
- class pysmi.searcher.pypackage.PyPackageSearcher(package)
Figures out if given Python module (source or bytecode) exists in given Python package.
Python package must be importable.
Create an instance of PyPackageSearcher bound to specific Python package.
- Parameters
package (str) -- name of the Python package to look up Python modules at.
Foundation or obsolete MIBs that should never be transformed can be blindly excluded by listing their names at the StubSearcher class instance.
- class pysmi.searcher.stub.StubSearcher(*mibnames)
Figures out if given MIB module is present in a fixed list of modules.
Create an instance of StubSearcher initialized with a fixed list or MIB modules names.
- Parameters
mibnames (str) -- blacklisted MIB names
NOTE:
A pysnmp-specific list of MIB names to be permanently excluded from transformation can be found at pysmi.codegen.pysnmp.baseMibs.
MIBs may be written in one of the two major SMI language versions (v1 and v2). Some MIBs may contain typical errors.
PySMI offers a way to customize the parser to consume either of the major SMI grammars as well as to recover from well-known errors in MIB files.
SNMP MIBs are written in two kinds of special language - SMIv1 and SMIv2. The first SMI version is obsolete, most MIBs by now are written in SMIv2 grammar. There are also efforts aimed at improving SMIv2, but those MIBs are in great minority at the time of this writing.
PySMI is designed to handle both SMIv1 and SMIv2. The way it is done is that SMIv2 is considered the most basic and complete, whereas SMIv1 is a specialization of SMIv2 syntax.
For a user to acquire SMIv2 parser the parserFactory function should be called with the SMI dialect object.
The parser object should be passed to the MibCompiler object.
- pysmi.parser.smi.parserFactory(**grammarOptions)
Factory function producing custom specializations of base SmiV2Parser class.
- Keyword Arguments
grammarOptions -- a list of (bool) typed optional keyword parameters enabling particular set of SMIv2 grammar relaxations.
- Returns
Specialized copy of SmiV2Parser class.
Notes
The following SMIv2 grammar relaxation parameters are defined:
supportSmiV1Keywords - parses SMIv1 grammar
supportIndex - tolerates ASN.1 types in INDEX clause
commaAtTheEndOfImport - tolerates stray comma at the end of IMPORT section
commaAtTheEndOfSequence - tolerates stray comma at the end of sequence of elements in MIB
mixOfCommasAndSpaces - tolerate a mix of comma and spaces in MIB enumerations
uppercaseIdentifier - tolerate uppercased MIB identifiers
lowcaseIdentifier - tolerate lowercase MIB identifiers
curlyBracesAroundEnterpriseInTrap - tolerate curly braces around enterprise ID in TRAP MACRO
noCells - tolerate missing cells (XXX)
Examples:
>>> from pysmi.parser import smi >>> SmiV1Parser = smi.parserFactory(supportSmiV1Keywords=True, supportIndex=True)
NOTE:
Please, note that parserFactory function returns a class, not class instance. Make sure to instantiate it when passing to MibCompiler class constructor.
PySMI offers a pre-built collection of parser grammar relaxation options to simplify its use:
pysmi.parser.dialect.smiV2 - canonical SMIv2 grammar
pysmi.parser.dialect.smiV1 - canonical SMIv1 grammar
pysmi.parser.dialect.smiV1Relaxed - relaxed SMIv1 grammar allowing some deviations
The grammar object should be passed to the parserFactory function.
from pysmi.parser.dialect import smiV1 from pysmi.parser.smi import parserFactory SmiV1Parser = parserFactory(**smiV1)
Apparently, many production MIBs were shipped in syntactically broken condition. PySMI attempts to work around such issues by allowing some extra SMI grammar relaxations. You can enable all those relaxations at once to maximize the number of MIBs, found in the wild, successfully compiled.
from pysmi.parser.dialect import smiV1Relaxed from pysmi.parser.smi import parserFactory RelaxedSmiV1Parser = parserFactory(**smiV1Relaxed)
Once ASN.1 MIB is parsed up, AST is passed to a code generator which turns AST into desired representation of the MIB.
- class pysmi.codegen.jsondoc.JsonCodeGen
Builds JSON document representing MIB module supplied in form of an Abstract Syntax Tree on input.
Instance of this class is supposed to be passed to MibCompiler, the rest is internal to MibCompiler.
- class pysmi.codegen.pysnmp.PySnmpCodeGen
Builds PySNMP-specific Python code representing MIB module supplied in form of an Abstract Syntax Tree on input.
Instance of this class is supposed to be passed to MibCompiler, the rest is internal to MibCompiler.
- class pysmi.codegen.null.NullCodeGen
Dummy code generation backend.
Could be used for disabling code generation at MibCompiler.
Some MIBs in circulation appear broken beyond automatic repair. To handle such cases PySMI introduces the MIB borrowing functionality. When MibCompiler gives up compiling a MIB, it can try to go out and take a copy of already transformed MIB to complete the request successfully.
- class pysmi.borrower.anyfile.AnyFileBorrower(reader, genTexts=False)
Create arbitrary MIB file borrowing object
Creates an instance of Borrower class.
- Parameters
reader -- a reader object
- Keyword Arguments
genText -- indicates whether this borrower should be looking for transformed MIBs that include human-oriented texts
- class pysmi.borrower.pyfile.PyFileBorrower(reader, genTexts=False)
Create PySNMP MIB file borrowing object
Creates an instance of Borrower class.
- Parameters
reader -- a reader object
- Keyword Arguments
genText -- indicates whether this borrower should be looking for transformed MIBs that include human-oriented texts
Successfully transformed MIB modules' contents will be passed to writer object given to MibCompiler on instantiation.
- class pysmi.writer.localfile.FileWriter(path)
Stores transformed MIB modules in files at specified location.
User is expected to pass FileReader class instance to MibCompiler on instantiation. The rest is internal to MibCompiler.
Creates an instance of FileReader class.
- Parameters
path -- writable directory to store created files
- class pysmi.writer.pyfile.PyFileWriter(path)
Stores transformed MIB modules as Python files at specified location.
User is expected to pass PyFileWriter class instance to MibCompiler on instantiation. The rest is internal to MibCompiler.
Creates an instance of PyFileWriter class.
- Parameters
path -- writable directory to store Python modules
CallbackWriter class instance passes the contents of the compiled MIB files to a user object.
- class pysmi.writer.callback.CallbackWriter(cbFun, cbCtx=None)
Invokes user-specified callable and passes transformed MIB module to it.
Note: user callable object signature must be as follows
- cbFun(mibname, contents, cbCtx)
Creates an instance of CallbackWriter class.
- Parameters
cbFun (callable) -- user-supplied callable
- Keyword Arguments
cbCtx -- user-supplied object passed intact to user callback
The following examples focus on various feature of the PySMI library.
Look up specific ASN.1 MIBs at configured Web and FTP sites, compile them into JSON documents and print them out to stdout.
Try to support both SMIv1 and SMIv2 flavors of SMI as well as popular deviations from official syntax found in the wild.
from pysmi.reader import FileReader, HttpReader from pysmi.searcher import StubSearcher from pysmi.writer import CallbackWriter from pysmi.parser import SmiStarParser from pysmi.codegen import JsonCodeGen from pysmi.compiler import MibCompiler # from pysmi import debug # debug.setLogger(debug.Debug('reader', 'compiler')) inputMibs = ['IF-MIB', 'IP-MIB'] srcDirectories = ['/usr/share/snmp/mibs'] httpSources = [ ('mibs.snmplabs.com', 80, '/asn1/@mib@') ] def printOut(mibName, jsonDoc, cbCtx): print('\n\n# MIB module %s' % mibName) print(jsonDoc) # Initialize compiler infrastructure mibCompiler = MibCompiler( SmiStarParser(), JsonCodeGen(), CallbackWriter(printOut) ) # search for source MIBs here mibCompiler.addSources(*[FileReader(x) for x in srcDirectories]) # search for source MIBs at Web sites mibCompiler.addSources(*[HttpReader(*x) for x in httpSources]) # never recompile MIBs with MACROs mibCompiler.addSearchers(StubSearcher(*JsonCodeGen.baseMibs)) # run recursive MIB compilation results = mibCompiler.compile(*inputMibs) print('\n# Results: %s' % ', '.join(['%s:%s' % (x, results[x]) for x in results]))
Download script.
Look up specific ASN.1 MIBs at configured Web and FTP sites, compile them into pysnmp form and save Python modules as plain-text files in a local directory.
Try to support both SMIv1 and SMIv2 flavors of SMI as well as popular deviations from official syntax found in the wild.
In this example we disable automatic dependency checking on MIB compilation using noDeps flag.
Also, we do not check if target file already exists thus MIB compilation occurs on every invocation.
from pysmi.reader import HttpReader from pysmi.reader import FtpReader from pysmi.searcher import StubSearcher from pysmi.writer import PyFileWriter from pysmi.parser import SmiStarParser from pysmi.codegen import PySnmpCodeGen from pysmi.compiler import MibCompiler inputMibs = ['IF-MIB', 'IP-MIB'] httpSources = [ ('mibs.snmplabs.com', 80, '/asn1/@mib@') ] ftpSources = [ ('ftp.cisco.com', '/pub/mibs/v2/@mib@') ] dstDirectory = '.pysnmp-mibs' # Initialize compiler infrastructure mibCompiler = MibCompiler( SmiStarParser(), PySnmpCodeGen(), PyFileWriter(dstDirectory) ) # search for source MIBs at Web and FTP sites mibCompiler.addSources(*[HttpReader(*x) for x in httpSources]) mibCompiler.addSources(*[FtpReader(*x) for x in ftpSources]) # never recompile MIBs with MACROs mibCompiler.addSearchers(StubSearcher(*PySnmpCodeGen.baseMibs)) # run non-recursive MIB compilation results = mibCompiler.compile(*inputMibs, **dict(noDeps=True)) print('Results: %s' % ', '.join(['%s:%s' % (x, results[x]) for x in results]))
Download script.
Look up specific ASN.1 MIBs at configured local directories, compile them into pysnmp form if not done yet and save Python modules as plain-text files in a local directory.
Try to support both SMIv1 and SMIv2 flavors of SMI as well as popular deviations from official syntax found in the wild.
When figuring out if compilation is needed, check all known places where pysnmp MIBs could possibly be found.
You can force MIB re-compilation by passing rebuild flag to MIB compiler (see below).
Default invocation of MIB compiler does not generate [potentially large] comments and texts found in MIBs. If you need them in pysnmp MIB modules, just pass genTexts flag to MIB compiler.
from pysmi.reader import FileReader from pysmi.searcher import PyFileSearcher, PyPackageSearcher, StubSearcher from pysmi.writer import PyFileWriter from pysmi.parser import SmiStarParser from pysmi.codegen import PySnmpCodeGen from pysmi.compiler import MibCompiler inputMibs = ['IF-MIB', 'IP-MIB'] srcDirectories = ['/usr/share/snmp/mibs'] dstDirectory = '.pysnmp-mibs' # Initialize compiler infrastructure mibCompiler = MibCompiler(SmiStarParser(), PySnmpCodeGen(), PyFileWriter(dstDirectory)) # search for source MIBs here mibCompiler.addSources(*[FileReader(x) for x in srcDirectories]) # check compiled MIBs in our own productions mibCompiler.addSearchers(PyFileSearcher(dstDirectory)) # ...and at default PySNMP MIBs packages mibCompiler.addSearchers(*[PyPackageSearcher(x) for x in PySnmpCodeGen.defaultMibPackages]) # never recompile MIBs with MACROs mibCompiler.addSearchers(StubSearcher(*PySnmpCodeGen.baseMibs)) # run [possibly recursive] MIB compilation results = mibCompiler.compile(*inputMibs) #, rebuild=True, genTexts=True) print('Results: %s' % ', '.join(['%s:%s' % (x, results[x]) for x in results]))
Download script.
Invoke user callback function to provide MIB text, compile given text string into pysnmp MIB form and pass results to another user callback function for storing.
Here we expect to deal only with SMIv2-valid MIBs.
We use noDeps flag to prevent MIB compiler from attemping to compile IMPORT'ed MIBs as well.
import sys from pysmi.reader import CallbackReader from pysmi.searcher import StubSearcher from pysmi.writer import CallbackWriter from pysmi.parser import SmiV2Parser from pysmi.codegen import PySnmpCodeGen from pysmi.compiler import MibCompiler inputMibs = ['IF-MIB', 'IP-MIB'] srcDir = '/usr/share/snmp/mibs/' # we will read MIBs from here # Initialize compiler infrastructure mibCompiler = MibCompiler( SmiV2Parser(), PySnmpCodeGen(), # out own callback function stores results in its own way CallbackWriter(lambda m, d, c: sys.stdout.write(d)) ) # our own callback function serves as a MIB source here mibCompiler.addSources( CallbackReader(lambda m, c: open(srcDir+m+'.txt').read()) ) # never recompile MIBs with MACROs mibCompiler.addSearchers(StubSearcher(*PySnmpCodeGen.baseMibs)) # run non-recursive MIB compilation results = mibCompiler.compile(*inputMibs, **dict(noDeps=True)) print('Results: %s' % ', '.join(['%s:%s' % (x, results[x]) for x in results]))
Download script.
Look up specific ASN.1 MIBs at configured Web/FTP sites. If no required MIB is found or its compilation fails for some reason, attempt to download precompiled version of failed MIB and store it locally as if we had compiled it.
from pysmi.reader import HttpReader from pysmi.searcher import PyFileSearcher from pysmi.searcher import StubSearcher from pysmi.borrower import PyFileBorrower from pysmi.writer import PyFileWriter from pysmi.parser import SmiStarParser from pysmi.codegen import PySnmpCodeGen from pysmi.compiler import MibCompiler # from pysmi import debug # debug.setLogger(debug.Debug('borrower', 'reader', 'searcher')) inputMibs = ['BORROWED-MIB'] httpSources = [ ('mibs.snmplabs.com', 80, '/asn1/@mib@') ] httpBorrowers = [ ('mibs.snmplabs.com', 80, '/pysnmp/notexts/@mib@') ] dstDirectory = '.pysnmp-mibs' # Initialize compiler infrastructure mibCompiler = MibCompiler( SmiStarParser(), PySnmpCodeGen(), PyFileWriter(dstDirectory) ) # search for source MIBs at Web sites mibCompiler.addSources(*[HttpReader(*x) for x in httpSources]) # never recompile MIBs with MACROs mibCompiler.addSearchers(StubSearcher(*PySnmpCodeGen.baseMibs)) # check compiled/borrowed MIBs in our own productions mibCompiler.addSearchers(PyFileSearcher(dstDirectory)) # search for compiled MIBs at Web sites if source is not available or broken mibCompiler.addBorrowers(*[PyFileBorrower(HttpReader(*x)).setOptions(genTexts=False) for x in httpBorrowers]) # run non-recursive MIB compilation results = mibCompiler.compile(*inputMibs) print('Results: %s' % ', '.join(['%s:%s' % (x, results[x]) for x in results]))
Download script.
Try to borrow precompiled pysnmp MIB file(s) from a web-site.
In this example no attempt is made to find and compile ASN.1 MIB source.
Fetched pysnmp MIB(s) are stored in a local directory.
from pysmi.reader import HttpReader from pysmi.searcher import PyFileSearcher from pysmi.borrower import PyFileBorrower from pysmi.writer import PyFileWriter from pysmi.parser import NullParser from pysmi.codegen import NullCodeGen from pysmi.compiler import MibCompiler inputMibs = ['BORROWED-MIB'] httpBorrowers = [ ('mibs.snmplabs.com', 80, '/pysnmp/notexts/@mib@') ] dstDirectory = '.pysnmp-mibs' # Initialize compiler infrastructure mibCompiler = MibCompiler( NullParser(), NullCodeGen(), PyFileWriter(dstDirectory) ) # check compiled/borrowed MIBs in our own productions mibCompiler.addSearchers(PyFileSearcher(dstDirectory)) # search for precompiled MIBs at Web sites mibCompiler.addBorrowers( *[PyFileBorrower(HttpReader(*x)) for x in httpBorrowers] ) # run MIB compilation results = mibCompiler.compile(*inputMibs) print('Results: %s' % ', '.join(['%s:%s' % (x, results[x]) for x in results]))
Download script.
In case of any troubles or confusion, try enabling PySMI debugging and watch the output:
from pysmi import debug debug.setLogger(debug.Debug('all'))
Project source code is hosted at GitHub. Everyone is welcome to fork and contribute back!
We maintain the detailed log of changes to our software.
The best way to obtain SNMP SMI library is by running pip:
$ virtualenv venv $ source venv/bin/activate $ pip install pysmi
Alternatively, you can download the latest release from GitHub or PyPI.
Added implied key to JSON SNMP table index structure
Rebased MIB importing code onto importlib because imp is long deprecated
Fixed Py file borrower to become functional
Added mibcopy.py documentation
Copyright notice bumped up to year 2019
Bumped upper Python version to 3.7 and enabled pip cache
Exit code indication of the command-line tools aligned with sysexits.h to report more useful termination status
Fixed pysnmp lower version in test-requirements.txt
Fixed compiler crash when building comments at a platform which has broken users/groups databases
The mibcopy tool implemented to copy MIB modules from files with potentially messed up names into a directory under canonical MIB names picking up the latest MIB revision along the way.
ZIP archive reader implemented to pull ASN.1 MIB files from .zip archives pretty much in the same way as from plain directories
HTTP/S proxy support added (through respecting http_proxy environment variable) by switching from httplib to urllib2 internally
Copyright notice bumped up to year 2018
Project site in the docs changes from SourceForge to snmplabs.com
PRODUCT-RELEASE generation added to the JSON code generator
Added special handling of BITS-like DEFVAL syntax for Integers that occurs in buggy MIBs
Fixed missing REVISIONS generations in MODULE-IDENTITY
Library documentation refactored and updated
Fixed malformed Python code being produced by pysnmp code generator
Added MIB status, product release and revision description set calls at pysnmp code generator
Changed REVISION field format in JSON representation - it is now a list of dicts each with revision timestamp and description text
MIB REFERENCE fields are only exported if --with-mib-text is on
Sphinx documentation theme changed to Alabaster
Multiple fixes to pysnmp codegen not to produce function calls with more than 255 parameters
Fix to SMI lexer to treat tokens starting from a digit as belonging to a lower-cased class. This fixes sub-OID parsing bug (specifically, 802dot3(10006))
Fix to the mibdump.py local MIB path automatic injection in front of existing --mib-sources
INET-ADDRESS-MIB configured as pre-built at pysnmp codegen
JSON codegen produces "nodetype" element for OBJECT-TYPE
Fix to mibdump.py --destination-directory option
Fix to pysnmp and JSON code generators to properly refer to MIB module defining particular MIB object
The @mib@ magic in reader's URL template made optional. If it is not present, MIB module name is just appended to URL template
Send User-Agent containing pysmi and Python versions as well as platform name.
Fixed missing STATUS/DISPLAY-HINT/REFERENCE/etc fields generation at pysnmp backend when running in the non-full-text mode
Fixed broken ordereddict dependency on Python 2.6-
Generate REFERENCE and STATUS fields at various SMI objects
Generate DESCRIPTION field followed REVISION field at MODULE-IDENTITY objects
Generate PRODUCT-RELEASE field at AGENT-CAPABILITIES objects
Generated Python source aligned with PEP8
MIB texts cleaned up by default, --keep-texts-layout preserves original formatting
Fix to the ordereddict conditional dependency
Missing test module recovered
Failing tests fixed
JSON code generating backend implemented
Experimental JSON OID->MIB indices generation implemented
Package structure flattened for easier use
Minor refactoring to the test suite
Source code statically analyzed, hardened and PEP8-ized
Files closed explicitly to mute ResourceWarnings
Fixed to Python 2.4 (and aged ply) compatibility
Added a workaround to avoid generating pysnmp TextualConvention classes inheriting from TextualConvention (when MIB defines a TEXTUAL-CONVENTION based on another TEXTUAL-CONVENTION as SYNTAX)
Author's e-mail changed, copyright extended to year 2017
Crash on existing .py file handling fixed.
Fix to __doc__ use in setup.py to make -O0 installation mode working.
Fix to PyPackageSearcher not to fail on broken Python packages.
Source code pep8'ed
Copyright added to source files.
Several typos fixed, source code linted again.
Some dead code cleaned up.
Wheel distribution format now supported.
Handle the case of MIB symbols conflict with Python reserved words.
Handle binary DEFVAL initializer for INTEGER's.
Generate LAST-UPDATED at pysnmp code generator.
Fix to MRO compliance for mixin classes generation at pysnmp backend
Fix to repeated imports in generated code at pysnmp backend
Fix to mibdump tool to properly handle the --generate-mib-texts option.
Fix to Python compile() - optimize flag is valid only past Python 3.1
Fix to SMIv1 INDEX clause code generation for pysnmp backend.
Tighten file creation security at pysmi.writer.pyfile
Two-pass compiler design allows for much accurate code generation.
Sphinx-based documentation first introduced
First public release, not fully operational yet
The SNMP SMI library software is distributed under 2-clause BSD License.
Copyright (c) 2015-2019 Ilya Etingof <etingof@gmail.com> All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The PySMI project maintains a collection of publicly available ASN.1 MIB files collected on the Internet. You are welcome to use this MIBs archive however we can't guarantee any degree of consistency or reliability when it comes to these MIB modules.
The mibdump.py tool as well as many other utilities based on PySMI are programmed to use this MIB repository for automatic download and dependency resolution.
You can always reconfigure PySMI to use some other remote MIB repository instead or in addition to this one.
In case of questions or troubles using SNMP SMI library, please open up an issue at GitHub or ask at Stack Overflow .
Ilya Etingof <etingof@gmail.com>
2015-2023, Ilya Etingof <etingof@gmail.com>