IPython/core/ultratb.py

# -*- coding: utf-8 -*-
"""
Verbose and colourful traceback formatting.

**ColorTB**

I've always found it a bit hard to visually parse tracebacks in Python.  The
ColorTB class is a solution to that problem.  It colors the different parts of a
traceback in a manner similar to what you would expect from a syntax-highlighting
text editor.

Installation instructions for ColorTB::

    import sys,ultratb
    sys.excepthook = ultratb.ColorTB()

**VerboseTB**

I've also included a port of Ka-Ping Yee's "cgitb.py" that produces all kinds
of useful info when a traceback occurs.  Ping originally had it spit out HTML
and intended it for CGI programmers, but why should they have all the fun?  I
altered it to spit out colored text to the terminal.  It's a bit overwhelming,
but kind of neat, and maybe useful for long-running programs that you believe
are bug-free.  If a crash *does* occur in that type of program you want details.
Give it a shot--you'll love it or you'll hate it.

.. note::

  The Verbose mode prints the variables currently visible where the exception
  happened (shortening their strings if too long). This can potentially be
  very slow, if you happen to have a huge data structure whose string
  representation is complex to compute. Your computer may appear to freeze for
  a while with cpu usage at 100%. If this occurs, you can cancel the traceback
  with Ctrl-C (maybe hitting it more than once).

  If you encounter this kind of situation often, you may want to use the
  Verbose_novars mode instead of the regular Verbose, which avoids formatting
  variables (but otherwise includes the information and context given by
  Verbose).

.. note::

  The verbose mode print all variables in the stack, which means it can
  potentially leak sensitive information like access keys, or unencryted
  password.

Installation instructions for VerboseTB::

    import sys,ultratb
    sys.excepthook = ultratb.VerboseTB()

Note:  Much of the code in this module was lifted verbatim from the standard
library module 'traceback.py' and Ka-Ping Yee's 'cgitb.py'.

Color schemes
-------------

The colors are defined in the class TBTools through the use of the
ColorSchemeTable class. Currently the following exist:

  - NoColor: allows all of this module to be used in any terminal (the color
    escapes are just dummy blank strings).

  - Linux: is meant to look good in a terminal like the Linux console (black
    or very dark background).

  - LightBG: similar to Linux but swaps dark/light colors to be more readable
    in light background terminals.

  - Neutral: a neutral color scheme that should be readable on both light and
    dark background

You can implement other color schemes easily, the syntax is fairly
self-explanatory. Please send back new schemes you develop to the author for
possible inclusion in future releases.

Inheritance diagram:

.. inheritance-diagram:: IPython.core.ultratb
   :parts: 3
"""

#*****************************************************************************
# Copyright (C) 2001 Nathaniel Gray <[e-mail]>
# Copyright (C) 2001-2004 Fernando Perez <[e-mail]>
#
# Distributed under the terms of the BSD License.  The full license is in
# the file COPYING, distributed as part of this software.
#*****************************************************************************

from __future__ import absolute_import
from __future__ import unicode_literals
from __future__ import print_function

import dis
import inspect
import keyword
import linecache
import os
import pydoc
import re
import sys
import time
import tokenize
import traceback
import types

try:  # Python 2
    generate_tokens = tokenize.generate_tokens
except AttributeError:  # Python 3
    generate_tokens = tokenize.tokenize

# For purposes of monkeypatching inspect to fix a bug in it.
from inspect import getsourcefile, getfile, getmodule, \
    ismodule, isclass, ismethod, isfunction, istraceback, isframe, iscode

# IPython's own modules
from IPython import get_ipython
from IPython.core import debugger
from IPython.core.display_trap import DisplayTrap
from IPython.core.excolors import exception_colors
from IPython.utils import PyColorize
from IPython.utils import openpy
from IPython.utils import path as util_path
from IPython.utils import py3compat
from IPython.utils import ulinecache
from IPython.utils.data import uniq_stable
from IPython.utils.terminal import get_terminal_size
from logging import info, error

import IPython.utils.colorable as colorable

# Globals
# amount of space to put line numbers before verbose tracebacks
INDENT_SIZE = 8

# Default color scheme.  This is used, for example, by the traceback
# formatter.  When running in an actual IPython instance, the user's rc.colors
# value is used, but having a module global makes this functionality available
# to users of ultratb who are NOT running inside ipython.
DEFAULT_SCHEME = 'NoColor'

# ---------------------------------------------------------------------------
# Code begins

# Utility functions
def inspect_error():
    """Print a message about internal inspect errors.

    These are unfortunately quite common."""

    error('Internal Python error in the inspect module.\n'
          'Below is the traceback from this internal error.\n')


# This function is a monkeypatch we apply to the Python inspect module. We have
# now found when it's needed (see discussion on issue gh-1456), and we have a
# test case (IPython.core.tests.test_ultratb.ChangedPyFileTest) that fails if
# the monkeypatch is not applied. TK, Aug 2012.
def findsource(object):
    """Return the entire source file and starting line number for an object.

    The argument may be a module, class, method, function, traceback, frame,
    or code object.  The source code is returned as a list of all the lines
    in the file and the line number indexes a line in that list.  An IOError
    is raised if the source code cannot be retrieved.

    FIXED version with which we monkeypatch the stdlib to work around a bug."""

    file = getsourcefile(object) or getfile(object)
    # If the object is a frame, then trying to get the globals dict from its
    # module won't work. Instead, the frame object itself has the globals
    # dictionary.
    globals_dict = None
    if inspect.isframe(object):
        # XXX: can this ever be false?
        globals_dict = object.f_globals
    else:
        module = getmodule(object, file)
        if module:
            globals_dict = module.__dict__
    lines = linecache.getlines(file, globals_dict)
    if not lines:
        raise IOError('could not get source code')

    if ismodule(object):
        return lines, 0

    if isclass(object):
        name = object.__name__
        pat = re.compile(r'^(\s*)class\s*' + name + r'\b')
        # make some effort to find the best matching class definition:
        # use the one with the least indentation, which is the one
        # that's most probably not inside a function definition.
        candidates = []
        for i in range(len(lines)):
            match = pat.match(lines[i])
            if match:
                # if it's at toplevel, it's already the best one
                if lines[i][0] == 'c':
                    return lines, i
                # else add whitespace to candidate list
                candidates.append((match.group(1), i))
        if candidates:
            # this will sort by whitespace, and by line number,
            # less whitespace first
            candidates.sort()
            return lines, candidates[0][1]
        else:
            raise IOError('could not find class definition')

    if ismethod(object):
        object = object.__func__
    if isfunction(object):
        object = object.__code__
    if istraceback(object):
        object = object.tb_frame
    if isframe(object):
        object = object.f_code
    if iscode(object):
        if not hasattr(object, 'co_firstlineno'):
            raise IOError('could not find function definition')
        pat = re.compile(r'^(\s*def\s)|(.*(?<!\w)lambda(:|\s))|^(\s*@)')
        pmatch = pat.match
        # fperez - fix: sometimes, co_firstlineno can give a number larger than
        # the length of lines, which causes an error.  Safeguard against that.
        lnum = min(object.co_firstlineno, len(lines)) - 1
        while lnum > 0:
            if pmatch(lines[lnum]):
                break
            lnum -= 1

        return lines, lnum
    raise IOError('could not find code object')


# This is a patched version of inspect.getargs that applies the (unmerged)
# patch for http://bugs.python.org/issue14611 by Stefano Taschini.  This fixes
# https://github.com/ipython/ipython/issues/8205 and
# https://github.com/ipython/ipython/issues/8293
def getargs(co):
    """Get information about the arguments accepted by a code object.

    Three things are returned: (args, varargs, varkw), where 'args' is
    a list of argument names (possibly containing nested lists), and
    'varargs' and 'varkw' are the names of the * and ** arguments or None."""
    if not iscode(co):
        raise TypeError('{!r} is not a code object'.format(co))

    nargs = co.co_argcount
    names = co.co_varnames
    args = list(names[:nargs])
    step = 0

    # The following acrobatics are for anonymous (tuple) arguments.
    for i in range(nargs):
        if args[i][:1] in ('', '.'):
            stack, remain, count = [], [], []
            while step < len(co.co_code):
                op = ord(co.co_code[step])
                step = step + 1
                if op >= dis.HAVE_ARGUMENT:
                    opname = dis.opname[op]
                    value = ord(co.co_code[step]) + ord(co.co_code[step+1])*256
                    step = step + 2
                    if opname in ('UNPACK_TUPLE', 'UNPACK_SEQUENCE'):
                        remain.append(value)
                        count.append(value)
                    elif opname in ('STORE_FAST', 'STORE_DEREF'):
                        if op in dis.haslocal:
                            stack.append(co.co_varnames[value])
                        elif op in dis.hasfree:
                            stack.append((co.co_cellvars + co.co_freevars)[value])
                        # Special case for sublists of length 1: def foo((bar))
                        # doesn't generate the UNPACK_TUPLE bytecode, so if
                        # `remain` is empty here, we have such a sublist.
                        if not remain:
                            stack[0] = [stack[0]]
                            break
                        else:
                            remain[-1] = remain[-1] - 1
                            while remain[-1] == 0:
                                remain.pop()
                                size = count.pop()
                                stack[-size:] = [stack[-size:]]
                                if not remain:
                                    break
                                remain[-1] = remain[-1] - 1
                            if not remain:
                                break
            args[i] = stack[0]

    varargs = None
    if co.co_flags & inspect.CO_VARARGS:
        varargs = co.co_varnames[nargs]
        nargs = nargs + 1
    varkw = None
    if co.co_flags & inspect.CO_VARKEYWORDS:
        varkw = co.co_varnames[nargs]
    return inspect.Arguments(args, varargs, varkw)


# Monkeypatch inspect to apply our bugfix.
def with_patch_inspect(f):
    """decorator for monkeypatching inspect.findsource"""

    def wrapped(*args, **kwargs):
        save_findsource = inspect.findsource
        save_getargs = inspect.getargs
        inspect.findsource = findsource
        inspect.getargs = getargs
        try:
            return f(*args, **kwargs)
        finally:
            inspect.findsource = save_findsource
            inspect.getargs = save_getargs

    return wrapped


if py3compat.PY3:
    fixed_getargvalues = inspect.getargvalues
else:
    # Fixes for https://github.com/ipython/ipython/issues/8293
    #       and https://github.com/ipython/ipython/issues/8205.
    # The relevant bug is caused by failure to correctly handle anonymous tuple
    # unpacking, which only exists in Python 2.
    fixed_getargvalues = with_patch_inspect(inspect.getargvalues)


def fix_frame_records_filenames(records):
    """Try to fix the filenames in each record from inspect.getinnerframes().

    Particularly, modules loaded from within zip files have useless filenames
    attached to their code object, and inspect.getinnerframes() just uses it.
    """
    fixed_records = []
    for frame, filename, line_no, func_name, lines, index in records:
        # Look inside the frame's globals dictionary for __file__,
        # which should be better. However, keep Cython filenames since
        # we prefer the source filenames over the compiled .so file.
        filename = py3compat.cast_unicode_py2(filename, "utf-8")
        if not filename.endswith(('.pyx', '.pxd', '.pxi')):
            better_fn = frame.f_globals.get('__file__', None)
            if isinstance(better_fn, str):
                # Check the type just in case someone did something weird with
                # __file__. It might also be None if the error occurred during
                # import.
                filename = better_fn
        fixed_records.append((frame, filename, line_no, func_name, lines, index))
    return fixed_records


@with_patch_inspect
def _fixed_getinnerframes(etb, context=1, tb_offset=0):
    LNUM_POS, LINES_POS, INDEX_POS = 2, 4, 5

    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
    # If the error is at the console, don't build any context, since it would
    # otherwise produce 5 blank lines printed out (there is no file at the
    # console)
    rec_check = records[tb_offset:]
    try:
        rname = rec_check[0][1]
        if rname == '<ipython console>' or rname.endswith('<string>'):
            return rec_check
    except IndexError:
        pass

    aux = traceback.extract_tb(etb)
    assert len(records) == len(aux)
    for i, (file, lnum, _, _) in zip(range(len(records)), aux):
        maybeStart = lnum - 1 - context // 2
        start = max(maybeStart, 0)
        end = start + context
        lines = ulinecache.getlines(file)[start:end]
        buf = list(records[i])
        buf[LNUM_POS] = lnum
        buf[INDEX_POS] = lnum - 1 - start
        buf[LINES_POS] = lines
        records[i] = tuple(buf)
    return records[tb_offset:]

# Helper function -- largely belongs to VerboseTB, but we need the same
# functionality to produce a pseudo verbose TB for SyntaxErrors, so that they
# can be recognized properly by ipython.el's py-traceback-line-re
# (SyntaxErrors have to be treated specially because they have no traceback)


def _format_traceback_lines(lnum, index, lines, Colors, lvals=None,  _line_format=(lambda x,_:x,None)):
    numbers_width = INDENT_SIZE - 1
    res = []
    i = lnum - index

    for line in lines:
        line = py3compat.cast_unicode(line)

        new_line, err = _line_format(line, 'str')
        if not err: line = new_line

        if i == lnum:
            # This is the line with the error
            pad = numbers_width - len(str(i))
            num = '%s%s' % (debugger.make_arrow(pad), str(lnum))
            line = '%s%s%s %s%s' % (Colors.linenoEm, num,
                                    Colors.line, line, Colors.Normal)
        else:
            num = '%*s' % (numbers_width, i)
            line = '%s%s%s %s' % (Colors.lineno, num,
                                  Colors.Normal, line)

        res.append(line)
        if lvals and i == lnum:
            res.append(lvals + '\n')
        i = i + 1
    return res

def is_recursion_error(etype, value, records):
    try:
        # RecursionError is new in Python 3.5
        recursion_error_type = RecursionError
    except NameError:
        recursion_error_type = RuntimeError

    # The default recursion limit is 1000, but some of that will be taken up
    # by stack frames in IPython itself. >500 frames probably indicates
    # a recursion error.
    return (etype is recursion_error_type) \
           and "recursion" in str(value).lower() \
           and len(records) > 500

def find_recursion(etype, value, records):
    """Identify the repeating stack frames from a RecursionError traceback

    'records' is a list as returned by VerboseTB.get_records()

    Returns (last_unique, repeat_length)
    """
    # This involves a bit of guesswork - we want to show enough of the traceback
    # to indicate where the recursion is occurring. We guess that the innermost
    # quarter of the traceback (250 frames by default) is repeats, and find the
    # first frame (from in to out) that looks different.
    if not is_recursion_error(etype, value, records):
        return len(records), 0

    # Select filename, lineno, func_name to track frames with
    records = [r[1:4] for r in records]
    inner_frames = records[-(len(records)//4):]
    frames_repeated = set(inner_frames)

    last_seen_at = {}
    longest_repeat = 0
    i = len(records)
    for frame in reversed(records):
        i -= 1
        if frame not in frames_repeated:
            last_unique = i
            break

        if frame in last_seen_at:
            distance = last_seen_at[frame] - i
            longest_repeat = max(longest_repeat, distance)

        last_seen_at[frame] = i
    else:
        last_unique = 0 # The whole traceback was recursion

    return last_unique, longest_repeat

#---------------------------------------------------------------------------
# Module classes
class TBTools(colorable.Colorable):
    """Basic tools used by all traceback printer classes."""

    # Number of frames to skip when reporting tracebacks
    tb_offset = 0

    def __init__(self, color_scheme='NoColor', call_pdb=False, ostream=None, parent=None, config=None):
        # Whether to call the interactive pdb debugger after printing
        # tracebacks or not
        super(TBTools, self).__init__(parent=parent, config=config)
        self.call_pdb = call_pdb

        # Output stream to write to.  Note that we store the original value in
        # a private attribute and then make the public ostream a property, so
        # that we can delay accessing sys.stdout until runtime.  The way
        # things are written now, the sys.stdout object is dynamically managed
        # so a reference to it should NEVER be stored statically.  This
        # property approach confines this detail to a single location, and all
        # subclasses can simply access self.ostream for writing.
        self._ostream = ostream

        # Create color table
        self.color_scheme_table = exception_colors()

        self.set_colors(color_scheme)
        self.old_scheme = color_scheme  # save initial value for toggles

        if call_pdb:
            self.pdb = debugger.Pdb()
        else:
            self.pdb = None

    def _get_ostream(self):
        """Output stream that exceptions are written to.

        Valid values are:

        - None: the default, which means that IPython will dynamically resolve
          to sys.stdout.  This ensures compatibility with most tools, including
          Windows (where plain stdout doesn't recognize ANSI escapes).

        - Any object with 'write' and 'flush' attributes.
        """
        return sys.stdout if self._ostream is None else self._ostream

    def _set_ostream(self, val):
        assert val is None or (hasattr(val, 'write') and hasattr(val, 'flush'))
        self._ostream = val

    ostream = property(_get_ostream, _set_ostream)

    def set_colors(self, *args, **kw):
        """Shorthand access to the color table scheme selector method."""

        # Set own color table
        self.color_scheme_table.set_active_scheme(*args, **kw)
        # for convenience, set Colors to the active scheme
        self.Colors = self.color_scheme_table.active_colors
        # Also set colors of debugger
        if hasattr(self, 'pdb') and self.pdb is not None:
            self.pdb.set_colors(*args, **kw)

    def color_toggle(self):
        """Toggle between the currently active color scheme and NoColor."""

        if self.color_scheme_table.active_scheme_name == 'NoColor':
            self.color_scheme_table.set_active_scheme(self.old_scheme)
            self.Colors = self.color_scheme_table.active_colors
        else:
            self.old_scheme = self.color_scheme_table.active_scheme_name
            self.color_scheme_table.set_active_scheme('NoColor')
            self.Colors = self.color_scheme_table.active_colors

    def stb2text(self, stb):
        """Convert a structured traceback (a list) to a string."""
        return '\n'.join(stb)

    def text(self, etype, value, tb, tb_offset=None, context=5):
        """Return formatted traceback.

        Subclasses may override this if they add extra arguments.
        """
        tb_list = self.structured_traceback(etype, value, tb,
                                            tb_offset, context)
        return self.stb2text(tb_list)

    def structured_traceback(self, etype, evalue, tb, tb_offset=None,
                             context=5, mode=None):
        """Return a list of traceback frames.

        Must be implemented by each class.
        """
        raise NotImplementedError()


#---------------------------------------------------------------------------
class ListTB(TBTools):
    """Print traceback information from a traceback list, with optional color.

    Calling requires 3 arguments: (etype, evalue, elist)
    as would be obtained by::
    
      etype, evalue, tb = sys.exc_info()
      if tb:
        elist = traceback.extract_tb(tb)
      else:
        elist = None

    It can thus be used by programs which need to process the traceback before
    printing (such as console replacements based on the code module from the
    standard library).

    Because they are meant to be called without a full traceback (only a
    list), instances of this class can't call the interactive pdb debugger."""

    def __init__(self, color_scheme='NoColor', call_pdb=False, ostream=None, parent=None, config=None):
        TBTools.__init__(self, color_scheme=color_scheme, call_pdb=call_pdb,
                         ostream=ostream, parent=parent,config=config)

    def __call__(self, etype, value, elist):
        self.ostream.flush()
        self.ostream.write(self.text(etype, value, elist))
        self.ostream.write('\n')

    def structured_traceback(self, etype, value, elist, tb_offset=None,
                             context=5):
        """Return a color formatted string with the traceback info.

        Parameters
        ----------
        etype : exception type
          Type of the exception raised.

        value : object
          Data stored in the exception

        elist : list
          List of frames, see class docstring for details.

        tb_offset : int, optional
          Number of frames in the traceback to skip.  If not given, the
          instance value is used (set in constructor).

        context : int, optional
          Number of lines of context information to print.

        Returns
        -------
        String with formatted exception.
        """
        tb_offset = self.tb_offset if tb_offset is None else tb_offset
        Colors = self.Colors
        out_list = []
        if elist:

            if tb_offset and len(elist) > tb_offset:
                elist = elist[tb_offset:]

            out_list.append('Traceback %s(most recent call last)%s:' %
                            (Colors.normalEm, Colors.Normal) + '\n')
            out_list.extend(self._format_list(elist))
        # The exception info should be a single entry in the list.
        lines = ''.join(self._format_exception_only(etype, value))
        out_list.append(lines)

        # Note: this code originally read:

        ## for line in lines[:-1]:
        ##     out_list.append(" "+line)
        ## out_list.append(lines[-1])

        # This means it was indenting everything but the last line by a little
        # bit.  I've disabled this for now, but if we see ugliness somewhere we
        # can restore it.

        return out_list

    def _format_list(self, extracted_list):
        """Format a list of traceback entry tuples for printing.

        Given a list of tuples as returned by extract_tb() or
        extract_stack(), return a list of strings ready for printing.
        Each string in the resulting list corresponds to the item with the
        same index in the argument list.  Each string ends in a newline;
        the strings may contain internal newlines as well, for those items
        whose source text line is not None.

        Lifted almost verbatim from traceback.py
        """

        Colors = self.Colors
        list = []
        for filename, lineno, name, line in extracted_list[:-1]:
            item = '  File %s"%s"%s, line %s%d%s, in %s%s%s\n' % \
                   (Colors.filename, py3compat.cast_unicode_py2(filename, "utf-8"), Colors.Normal,
                    Colors.lineno, lineno, Colors.Normal,
                    Colors.name, py3compat.cast_unicode_py2(name, "utf-8"), Colors.Normal)
            if line:
                item += '    %s\n' % line.strip()
            list.append(item)
        # Emphasize the last entry
        filename, lineno, name, line = extracted_list[-1]
        item = '%s  File %s"%s"%s, line %s%d%s, in %s%s%s%s\n' % \
               (Colors.normalEm,
                Colors.filenameEm, py3compat.cast_unicode_py2(filename, "utf-8"), Colors.normalEm,
                Colors.linenoEm, lineno, Colors.normalEm,
                Colors.nameEm, py3compat.cast_unicode_py2(name, "utf-8"), Colors.normalEm,
                Colors.Normal)
        if line:
            item += '%s    %s%s\n' % (Colors.line, line.strip(),
                                      Colors.Normal)
        list.append(item)
        return list

    def _format_exception_only(self, etype, value):
        """Format the exception part of a traceback.