2012-04-08 16:43:30 +08:00
# -*- coding: utf-8 -*-
2012-01-29 14:25:45 +08:00
2013-05-23 22:47:07 +08:00
# Run me like this: ./create_manpage_completions.py /usr/share/man/man{1,8}/* > man_completions.fish
2012-04-05 03:43:12 +08:00
2012-01-29 15:32:40 +08:00
"""
2012-11-18 18:23:22 +08:00
< OWNER > = Siteshwar Vashisht
< YEAR > = 2012
2012-01-29 15:32:40 +08:00
2012-11-18 18:23:22 +08:00
Copyright ( c ) 2012 , Siteshwar Vashisht
2012-01-29 15:32:40 +08:00
All rights reserved .
Redistribution and use in source and binary forms , with or without modification , are permitted provided that the following conditions are met :
Redistributions of source code must retain the above copyright notice , this list of conditions and the following disclaimer .
Redistributions in binary form must reproduce the above copyright notice , this list of conditions and the following disclaimer in the documentation and / or other materials provided with the distribution .
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS " AS IS " AND ANY EXPRESS OR IMPLIED WARRANTIES , INCLUDING , BUT NOT LIMITED TO , THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED . IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT , INDIRECT , INCIDENTAL , SPECIAL , EXEMPLARY , OR CONSEQUENTIAL DAMAGES ( INCLUDING , BUT NOT LIMITED TO , PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES ; LOSS OF USE , DATA , OR PROFITS ; OR BUSINESS INTERRUPTION ) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY , WHETHER IN CONTRACT , STRICT LIABILITY , OR TORT ( INCLUDING NEGLIGENCE OR OTHERWISE ) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE , EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE .
"""
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
from __future__ import print_function
2012-04-08 16:43:30 +08:00
from deroff import Deroffer
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
import argparse
import bz2
import codecs
import errno
import gzip
import os
import re
import string
import subprocess
import sys
import traceback
2012-03-16 18:03:43 +08:00
2013-09-25 20:36:42 +08:00
lzma_available = True
2013-09-24 21:42:32 +08:00
try :
2013-09-25 20:36:42 +08:00
try :
import lzma
2013-11-05 23:30:51 +08:00
except ImportError :
from backports import lzma
2013-09-25 20:35:11 +08:00
except ImportError :
2013-09-25 20:36:42 +08:00
lzma_available = False
2013-09-24 21:42:32 +08:00
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
try :
from subprocess import DEVNULL
except ImportError :
2020-06-25 02:43:56 +08:00
DEVNULL = open ( os . devnull , " wb " )
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
2012-06-19 04:59:07 +08:00
# Whether we're Python 3
IS_PY3 = sys . version_info [ 0 ] > = 3
2012-01-29 14:25:45 +08:00
# This gets set to the name of the command that we are currently executing
CMDNAME = " "
2012-06-19 04:59:07 +08:00
# Information used to track which of our parsers were successful
PARSER_INFO = { }
2013-02-18 08:14:36 +08:00
# built_command writes into this global variable, yuck
2012-03-31 01:00:01 +08:00
built_command_output = [ ]
2012-01-29 14:25:45 +08:00
2012-04-04 09:38:25 +08:00
# Diagnostic output
diagnostic_output = [ ]
diagnostic_indent = 0
2012-04-12 10:05:43 +08:00
# Three diagnostic verbosity levels
VERY_VERBOSE , BRIEF_VERBOSE , NOT_VERBOSE = 2 , 1 , 0
2012-04-05 03:43:12 +08:00
2012-04-12 10:05:43 +08:00
# Pick some reasonable default values for settings
2020-04-04 20:18:10 +08:00
global VERBOSITY , WRITE_TO_STDOUT , DEROFF_ONLY , KEEP_FILES
VERBOSITY , WRITE_TO_STDOUT , DEROFF_ONLY , KEEP_FILES = NOT_VERBOSE , False , False , False
2012-04-12 10:05:43 +08:00
2019-05-05 18:09:25 +08:00
def add_diagnostic ( dgn , msg_verbosity = VERY_VERBOSE ) :
2012-04-12 10:05:43 +08:00
# Add a diagnostic message, if msg_verbosity <= VERBOSITY
if msg_verbosity < = VERBOSITY :
2019-05-05 18:09:25 +08:00
diagnostic_output . append ( " " * diagnostic_indent + dgn )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
def flush_diagnostics ( where ) :
if diagnostic_output :
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
output_str = " \n " . join ( diagnostic_output )
print ( output_str , file = where )
2012-04-04 09:38:25 +08:00
diagnostic_output [ : ] = [ ]
2019-05-05 18:09:25 +08:00
2012-04-04 09:38:25 +08:00
# Make sure we don't output the same completion multiple times, which can happen
# For example, xsubpp.1.gz and xsubpp5.10.1.gz
# This maps commands to lists of completions
already_output_completions = { }
2019-05-05 18:09:25 +08:00
2013-07-06 23:27:41 +08:00
def compile_and_search ( regex , input ) :
2019-05-05 18:09:25 +08:00
options_section_regex = re . compile ( regex , re . DOTALL )
options_section_matched = re . search ( options_section_regex , input )
2012-04-04 09:38:25 +08:00
return options_section_matched
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
2013-07-06 23:27:41 +08:00
def unquote_double_quotes ( data ) :
2019-05-05 18:09:25 +08:00
if len ( data ) < 2 :
2012-04-04 09:38:25 +08:00
return data
2019-05-05 18:09:25 +08:00
if data [ 0 ] == ' " ' and data [ len ( data ) - 1 ] == ' " ' :
data = data [ 1 : len ( data ) - 1 ]
2012-04-04 09:38:25 +08:00
return data
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
2013-07-06 23:27:41 +08:00
def unquote_single_quotes ( data ) :
2019-05-05 18:09:25 +08:00
if len ( data ) < 2 :
2012-04-04 09:38:25 +08:00
return data
2019-05-05 18:09:25 +08:00
if data [ 0 ] == " ` " and data [ len ( data ) - 1 ] == " ' " :
data = data [ 1 : len ( data ) - 1 ]
2012-04-04 09:38:25 +08:00
return data
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
# Make a string of characters that are deemed safe in fish without needing to be escaped
# Note that space is not included
2019-05-05 18:09:25 +08:00
g_fish_safe_chars = frozenset ( string . ascii_letters + string . digits + " _+-|/:=@~ " )
2012-04-04 09:38:25 +08:00
def fish_escape_single_quote ( str ) :
# Escape a string if necessary so that it can be put in single quotes
# If it has no non-safe chars, there's nothing to do
if g_fish_safe_chars . issuperset ( str ) :
return str
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
str = str . replace ( " \\ " , " \\ \\ " ) # Replace one backslash with two
str = str . replace (
" ' " , " \\ ' "
) # Replace one single quote with a backslash-single-quote
2012-04-04 09:38:25 +08:00
return " ' " + str + " ' "
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
2013-04-15 16:05:37 +08:00
# Make a string Unicode by attempting to decode it as latin-1, or UTF8. See #658
def lossy_unicode ( s ) :
# All strings are unicode in Python 3
2019-05-05 18:09:25 +08:00
if IS_PY3 or isinstance ( s , unicode ) :
return s
2013-04-15 16:05:37 +08:00
try :
2019-05-05 18:09:25 +08:00
return s . decode ( " latin-1 " )
2013-04-15 16:05:37 +08:00
except UnicodeEncodeError :
pass
try :
2019-05-05 18:09:25 +08:00
return s . decode ( " utf-8 " )
2013-04-15 16:05:37 +08:00
except UnicodeEncodeError :
pass
2019-05-05 18:09:25 +08:00
return s . decode ( " latin-1 " , " ignore " )
2016-10-10 05:11:04 +08:00
2013-04-15 16:05:37 +08:00
2012-03-31 01:00:01 +08:00
def output_complete_command ( cmdname , args , description , output_list ) :
2019-05-05 18:09:25 +08:00
comps = [ " complete -c " , cmdname ]
2012-04-04 09:38:25 +08:00
comps . extend ( args )
if description :
2020-04-25 17:24:08 +08:00
comps . append ( " -d " )
2012-04-04 09:38:25 +08:00
comps . append ( description )
2019-05-05 18:09:25 +08:00
output_list . append ( lossy_unicode ( " " ) . join ( [ lossy_unicode ( c ) for c in comps ] ) )
2012-01-29 14:25:45 +08:00
2013-02-18 08:14:36 +08:00
def built_command ( options , description ) :
2019-05-05 18:09:25 +08:00
# print "Options are: ", options
man_optionlist = re . split ( ' |,| " |=|[|] ' , options )
2012-04-04 09:38:25 +08:00
fish_options = [ ]
2016-07-31 12:57:11 +08:00
for optionstr in man_optionlist :
option = re . sub ( r " ( \ [.* \ ]) " , " " , optionstr )
2017-08-12 02:51:28 +08:00
option = option . strip ( " \t \r \n []() {} .,:! " )
2016-07-31 12:57:11 +08:00
2012-04-09 14:26:26 +08:00
# Skip some problematic cases
2019-05-05 18:09:25 +08:00
if option in [ " - " , " -- " ] :
continue
if any ( c in " {} () " for c in option ) :
continue
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
if option . startswith ( " -- " ) :
2012-04-04 09:38:25 +08:00
# New style long option (--recursive)
2019-05-05 18:09:25 +08:00
fish_options . append ( " -l " + fish_escape_single_quote ( option [ 2 : ] ) )
elif option . startswith ( " - " ) and len ( option ) == 2 :
2012-04-04 09:38:25 +08:00
# New style short option (-r)
2019-05-05 18:09:25 +08:00
fish_options . append ( " -s " + fish_escape_single_quote ( option [ 1 : ] ) )
elif option . startswith ( " - " ) and len ( option ) > 2 :
2012-04-04 09:38:25 +08:00
# Old style long option (-recursive)
2019-05-05 18:09:25 +08:00
fish_options . append ( " -o " + fish_escape_single_quote ( option [ 1 : ] ) )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
# Determine which options are new (not already in existing_options)
# Then add those to the existing options
existing_options = already_output_completions . setdefault ( CMDNAME , set ( ) )
fish_options = [ opt for opt in fish_options if opt not in existing_options ]
existing_options . update ( fish_options )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
# Maybe it's all for naught
2019-05-05 18:09:25 +08:00
if not fish_options :
return
2012-11-18 18:23:22 +08:00
2012-11-23 12:32:19 +08:00
# Here's what we'll use to truncate if necessary
2016-07-09 20:28:03 +08:00
max_description_width = 78
2013-01-09 06:57:48 +08:00
if IS_PY3 :
2019-05-05 18:09:25 +08:00
truncation_suffix = " … "
2013-01-09 06:57:48 +08:00
else :
2013-04-14 16:58:34 +08:00
ELLIPSIS_CODE_POINT = 0x2026
2016-07-09 20:28:03 +08:00
truncation_suffix = unichr ( ELLIPSIS_CODE_POINT )
2016-10-10 05:11:04 +08:00
2012-11-23 12:32:19 +08:00
# Try to include as many whole sentences as will fit
2013-01-28 04:58:52 +08:00
# Clean up some probably bogus escapes in the process
clean_desc = description . replace ( " \\ ' " , " ' " ) . replace ( " \\ . " , " . " )
2019-05-05 18:09:25 +08:00
sentences = clean_desc . split ( " . " )
2016-10-10 05:11:04 +08:00
2013-01-28 04:58:52 +08:00
# Clean up "sentences" that are just whitespace
# But don't let it be empty
sentences = [ x for x in sentences if x . strip ( ) ]
2019-05-05 18:09:25 +08:00
if not sentences :
sentences = [ " " ]
2013-05-12 16:57:42 +08:00
2019-05-05 18:09:25 +08:00
udot = lossy_unicode ( " . " )
uspace = lossy_unicode ( " " )
2016-10-10 05:11:04 +08:00
truncated_description = lossy_unicode ( sentences [ 0 ] ) + udot
2012-11-23 12:32:19 +08:00
for line in sentences [ 1 : ] :
2019-05-05 18:09:25 +08:00
if not line :
continue
proposed_description = (
lossy_unicode ( truncated_description ) + uspace + lossy_unicode ( line ) + udot
)
2012-11-23 12:32:19 +08:00
if len ( proposed_description ) < = max_description_width :
# It fits
truncated_description = proposed_description
else :
# No fit
break
2020-04-25 17:22:41 +08:00
# Strip trailing dots
truncated_description = truncated_description . strip ( udot )
2016-10-10 05:11:04 +08:00
2012-11-23 12:32:19 +08:00
# If the first sentence does not fit, truncate if necessary
if len ( truncated_description ) > max_description_width :
prefix_len = max_description_width - len ( truncation_suffix )
truncated_description = truncated_description [ : prefix_len ] + truncation_suffix
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
# Escape some more things
2012-11-23 12:32:19 +08:00
truncated_description = fish_escape_single_quote ( truncated_description )
2012-04-04 09:38:25 +08:00
escaped_cmd = fish_escape_single_quote ( CMDNAME )
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
output_complete_command (
escaped_cmd , fish_options , truncated_description , built_command_output
)
2012-04-04 09:38:25 +08:00
2013-07-06 23:27:41 +08:00
def remove_groff_formatting ( data ) :
2019-05-05 18:09:25 +08:00
data = data . replace ( " \\ fI " , " " )
data = data . replace ( " \\ fP " , " " )
data = data . replace ( " \\ f1 " , " " )
data = data . replace ( " \\ fB " , " " )
data = data . replace ( " \\ fR " , " " )
data = data . replace ( " \\ e " , " " )
data = re . sub ( " .PD( \ d+) " , " " , data )
data = data . replace ( " .BI " , " " )
data = data . replace ( " .BR " , " " )
data = data . replace ( " 0.5i " , " " )
data = data . replace ( " .rb " , " " )
data = data . replace ( " \\ ^ " , " " )
data = data . replace ( " { " , " " )
data = data . replace ( " } " , " " )
data = data . replace ( " \ " , " " )
data = data . replace ( " \ - " , " - " )
data = data . replace ( " \ & " , " " )
data = data . replace ( " .B " , " " )
data = data . replace ( " \ - " , " - " )
data = data . replace ( " .I " , " " )
data = data . replace ( " \f " , " " )
2020-06-06 22:22:56 +08:00
data = data . replace ( " \ (cq " , " ' " )
2012-04-04 09:38:25 +08:00
return data
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
2016-07-09 20:28:03 +08:00
class ManParser ( object ) :
2013-07-06 23:27:41 +08:00
def is_my_type ( self , manpage ) :
2012-04-04 09:38:25 +08:00
return False
2012-01-29 14:25:45 +08:00
2013-07-06 23:27:41 +08:00
def parse_man_page ( self , manpage ) :
2012-04-04 09:38:25 +08:00
return False
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
2012-01-29 14:25:45 +08:00
class Type1ManParser ( ManParser ) :
2013-07-06 23:27:41 +08:00
def is_my_type ( self , manpage ) :
2020-07-09 18:20:09 +08:00
return compile_and_search ( ' \ .SH " OPTIONS " (.*?) ' , manpage ) != None
2012-04-04 09:38:25 +08:00
2013-07-06 23:27:41 +08:00
def parse_man_page ( self , manpage ) :
2019-05-05 18:09:25 +08:00
options_section_regex = re . compile ( ' \ .SH " OPTIONS " (.*?)( \ .SH| \ Z) ' , re . DOTALL )
2020-07-09 18:20:09 +08:00
options_section = re . search ( options_section_regex , manpage ) . group ( 1 )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
options_parts_regex = re . compile ( " \ .PP(.*?) \ .RE " , re . DOTALL )
options_matched = re . search ( options_parts_regex , options_section )
2016-07-09 20:28:03 +08:00
add_diagnostic ( " Command is %r " % CMDNAME )
2012-04-04 09:38:25 +08:00
if options_matched == None :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Unable to find options " )
if self . fallback ( options_section ) :
2012-04-04 09:38:25 +08:00
return True
2019-05-05 18:09:25 +08:00
elif self . fallback2 ( options_section ) :
2012-04-04 09:38:25 +08:00
return True
return False
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
while options_matched != None :
2012-04-04 09:38:25 +08:00
data = options_matched . group ( 1 )
last_dotpp_index = data . rfind ( " .PP " )
2019-05-05 18:09:25 +08:00
if last_dotpp_index != - 1 :
data = data [ last_dotpp_index + 3 : ]
2012-04-04 09:38:25 +08:00
2013-07-06 23:27:41 +08:00
data = remove_groff_formatting ( data )
2012-04-04 09:38:25 +08:00
data = data . split ( " .RS 4 " )
2019-05-05 18:09:25 +08:00
if len ( data ) > 1 : # and len(data[1]) <= 300):
2012-04-04 09:38:25 +08:00
optionName = data [ 0 ] . strip ( )
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
if optionName . find ( " - " ) == - 1 :
2016-07-09 20:28:03 +08:00
add_diagnostic ( " %r doesn ' t contain ' - ' " % optionName )
2012-04-04 09:38:25 +08:00
else :
2013-07-06 23:27:41 +08:00
optionName = unquote_double_quotes ( optionName )
optionName = unquote_single_quotes ( optionName )
2019-05-05 18:09:25 +08:00
optionDescription = data [ 1 ] . strip ( ) . replace ( " \n " , " " )
2013-02-18 08:14:36 +08:00
built_command ( optionName , optionDescription )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
else :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Unable to split option from description " )
2012-04-04 09:38:25 +08:00
return False
2019-05-05 18:09:25 +08:00
options_section = options_section [ options_matched . end ( ) - 3 : ]
2012-04-04 09:38:25 +08:00
options_matched = re . search ( options_parts_regex , options_section )
def fallback ( self , options_section ) :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Trying fallback " )
2012-04-04 09:38:25 +08:00
options_parts_regex = re . compile ( " \ .TP( \ d+)?(.*?) \ .TP " , re . DOTALL )
options_matched = re . search ( options_parts_regex , options_section )
if options_matched == None :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Still not found " )
2012-04-04 09:38:25 +08:00
return False
while options_matched != None :
data = options_matched . group ( 2 )
2013-07-06 23:27:41 +08:00
data = remove_groff_formatting ( data )
2012-04-04 09:38:25 +08:00
data = data . strip ( )
2019-05-05 18:09:25 +08:00
data = data . split ( " \n " , 1 )
if len ( data ) > 1 and len ( data [ 1 ] . strip ( ) ) > 0 : # and len(data[1])<400):
2012-04-04 09:38:25 +08:00
optionName = data [ 0 ] . strip ( )
2019-05-05 18:09:25 +08:00
if optionName . find ( " - " ) == - 1 :
2016-07-09 20:28:03 +08:00
add_diagnostic ( " %r doesn ' t contain ' - ' " % optionName )
2012-04-04 09:38:25 +08:00
else :
2013-07-06 23:27:41 +08:00
optionName = unquote_double_quotes ( optionName )
optionName = unquote_single_quotes ( optionName )
2019-05-05 18:09:25 +08:00
optionDescription = data [ 1 ] . strip ( ) . replace ( " \n " , " " )
2013-02-18 08:14:36 +08:00
built_command ( optionName , optionDescription )
2012-04-04 09:38:25 +08:00
else :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Unable to split option from description " )
2012-04-04 09:38:25 +08:00
return False
2019-05-05 18:09:25 +08:00
options_section = options_section [ options_matched . end ( ) - 3 : ]
2012-04-04 09:38:25 +08:00
options_matched = re . search ( options_parts_regex , options_section )
return True
def fallback2 ( self , options_section ) :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Trying last chance fallback " )
2012-04-04 09:38:25 +08:00
ix_remover_regex = re . compile ( " \ .IX.* " )
2019-05-05 18:09:25 +08:00
trailing_num_regex = re . compile ( " \\ d+$ " )
2012-04-04 09:38:25 +08:00
options_parts_regex = re . compile ( " \ .IP (.*?) \ .IP " , re . DOTALL )
options_section = re . sub ( ix_remover_regex , " " , options_section )
options_matched = re . search ( options_parts_regex , options_section )
if options_matched == None :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Still (still!) not found " )
2012-04-04 09:38:25 +08:00
return False
while options_matched != None :
data = options_matched . group ( 1 )
2012-11-18 18:23:22 +08:00
2013-07-06 23:27:41 +08:00
data = remove_groff_formatting ( data )
2012-04-04 09:38:25 +08:00
data = data . strip ( )
2019-05-05 18:09:25 +08:00
data = data . split ( " \n " , 1 )
if len ( data ) > 1 and len ( data [ 1 ] . strip ( ) ) > 0 : # and len(data[1])<400):
2012-04-04 09:38:25 +08:00
optionName = re . sub ( trailing_num_regex , " " , data [ 0 ] . strip ( ) )
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
if " - " not in optionName :
2016-07-09 20:28:03 +08:00
add_diagnostic ( " %r doesn ' t contain ' - ' " % optionName )
2012-04-04 09:38:25 +08:00
else :
optionName = optionName . strip ( )
2013-07-06 23:27:41 +08:00
optionName = unquote_double_quotes ( optionName )
optionName = unquote_single_quotes ( optionName )
2019-05-05 18:09:25 +08:00
optionDescription = data [ 1 ] . strip ( ) . replace ( " \n " , " " )
2013-02-18 08:14:36 +08:00
built_command ( optionName , optionDescription )
2012-04-04 09:38:25 +08:00
else :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Unable to split option from description " )
2012-04-04 09:38:25 +08:00
return False
2019-05-05 18:09:25 +08:00
options_section = options_section [ options_matched . end ( ) - 3 : ]
2012-04-04 09:38:25 +08:00
options_matched = re . search ( options_parts_regex , options_section )
return True
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
2012-01-29 14:25:45 +08:00
class Type2ManParser ( ManParser ) :
2013-07-06 23:27:41 +08:00
def is_my_type ( self , manpage ) :
2020-07-09 18:20:09 +08:00
return compile_and_search ( " \ .SH OPTIONS(.*?) " , manpage ) != None
2012-04-04 09:38:25 +08:00
2013-07-06 23:27:41 +08:00
def parse_man_page ( self , manpage ) :
2019-05-05 18:09:25 +08:00
options_section_regex = re . compile ( " \ .SH OPTIONS(.*?)( \ .SH| \ Z) " , re . DOTALL )
2020-07-09 18:20:09 +08:00
options_section = re . search ( options_section_regex , manpage ) . group ( 1 )
2012-04-04 09:38:25 +08:00
2020-11-22 21:39:48 +08:00
options_parts_regex = re . compile (
" \ .[IT]P( \ d+( \ . \ d)?i?)?(.*?) \ .([IT]P|UNINDENT|UN|SH) " , re . DOTALL
)
2012-04-04 09:38:25 +08:00
options_matched = re . search ( options_parts_regex , options_section )
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Command is %r " % CMDNAME )
2012-04-04 09:38:25 +08:00
if options_matched == None :
2016-07-09 20:28:03 +08:00
add_diagnostic ( " %r : Unable to find options " % self )
2012-04-04 09:38:25 +08:00
return False
2019-05-05 18:09:25 +08:00
while options_matched != None :
2012-04-04 09:38:25 +08:00
data = options_matched . group ( 3 )
2013-07-06 23:27:41 +08:00
data = remove_groff_formatting ( data )
2012-04-04 09:38:25 +08:00
data = data . strip ( )
2019-05-05 18:09:25 +08:00
data = data . split ( " \n " , 1 )
2020-07-09 18:20:09 +08:00
2019-05-05 18:09:25 +08:00
if len ( data ) > 1 and len ( data [ 1 ] . strip ( ) ) > 0 : # and len(data[1])<400):
2012-04-04 09:38:25 +08:00
optionName = data [ 0 ] . strip ( )
2019-05-05 18:09:25 +08:00
if " - " not in optionName :
2016-07-09 20:28:03 +08:00
add_diagnostic ( " %r doesn ' t contain ' - ' " % optionName )
2012-04-04 09:38:25 +08:00
else :
2013-07-06 23:27:41 +08:00
optionName = unquote_double_quotes ( optionName )
optionName = unquote_single_quotes ( optionName )
2019-05-05 18:09:25 +08:00
optionDescription = data [ 1 ] . strip ( ) . replace ( " \n " , " " )
2013-02-18 08:14:36 +08:00
built_command ( optionName , optionDescription )
2012-04-04 09:38:25 +08:00
else :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Unable to split option from description " )
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
options_section = options_section [ options_matched . end ( ) - 3 : ]
2012-04-04 09:38:25 +08:00
options_matched = re . search ( options_parts_regex , options_section )
2020-11-22 21:39:48 +08:00
2012-01-29 14:25:45 +08:00
class Type3ManParser ( ManParser ) :
2013-07-06 23:27:41 +08:00
def is_my_type ( self , manpage ) :
2020-07-09 18:20:09 +08:00
return compile_and_search ( " \ .SH DESCRIPTION(.*?) " , manpage ) != None
2012-01-29 14:25:45 +08:00
2013-07-06 23:27:41 +08:00
def parse_man_page ( self , manpage ) :
2019-05-05 18:09:25 +08:00
options_section_regex = re . compile ( " \ .SH DESCRIPTION(.*?)( \ .SH| \ Z) " , re . DOTALL )
2020-07-09 18:20:09 +08:00
options_section = re . search ( options_section_regex , manpage ) . group ( 1 )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
options_parts_regex = re . compile ( " \ .TP(.*?) \ .TP " , re . DOTALL )
options_matched = re . search ( options_parts_regex , options_section )
2016-07-09 20:28:03 +08:00
add_diagnostic ( " Command is %r " % CMDNAME )
2012-01-29 14:25:45 +08:00
2012-04-04 09:38:25 +08:00
if options_matched == None :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Unable to find options section " )
2012-04-04 09:38:25 +08:00
return False
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
while options_matched != None :
2012-04-04 09:38:25 +08:00
data = options_matched . group ( 1 )
2012-01-29 14:25:45 +08:00
2013-07-06 23:27:41 +08:00
data = remove_groff_formatting ( data )
2012-04-04 09:38:25 +08:00
data = data . strip ( )
2019-05-05 18:09:25 +08:00
data = data . split ( " \n " , 1 )
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
if len ( data ) > 1 : # and len(data[1])<400):
2012-04-04 09:38:25 +08:00
optionName = data [ 0 ] . strip ( )
2019-05-05 18:09:25 +08:00
if optionName . find ( " - " ) == - 1 :
2016-07-09 20:28:03 +08:00
add_diagnostic ( " %r doesn ' t contain ' - ' " % optionName )
2012-04-04 09:38:25 +08:00
else :
2013-07-06 23:27:41 +08:00
optionName = unquote_double_quotes ( optionName )
optionName = unquote_single_quotes ( optionName )
2019-05-05 18:09:25 +08:00
optionDescription = data [ 1 ] . strip ( ) . replace ( " \n " , " " )
2013-02-18 08:14:36 +08:00
built_command ( optionName , optionDescription )
2012-01-29 14:25:45 +08:00
2012-04-04 09:38:25 +08:00
else :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Unable to split option from description " )
2012-04-04 09:38:25 +08:00
return False
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
options_section = options_section [ options_matched . end ( ) - 3 : ]
2012-04-04 09:38:25 +08:00
options_matched = re . search ( options_parts_regex , options_section )
2012-01-29 14:25:45 +08:00
class Type4ManParser ( ManParser ) :
2013-07-06 23:27:41 +08:00
def is_my_type ( self , manpage ) :
2020-07-09 18:20:09 +08:00
return compile_and_search ( " \ .SH FUNCTION LETTERS(.*?) " , manpage ) != None
2012-04-04 09:38:25 +08:00
2013-07-06 23:27:41 +08:00
def parse_man_page ( self , manpage ) :
2020-11-22 21:39:48 +08:00
options_section_regex = re . compile (
" \ .SH FUNCTION LETTERS(.*?)( \ .SH| \ Z) " , re . DOTALL
)
2020-07-09 18:20:09 +08:00
options_section = re . search ( options_section_regex , manpage ) . group ( 1 )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
options_parts_regex = re . compile ( " \ .TP(.*?) \ .TP " , re . DOTALL )
options_matched = re . search ( options_parts_regex , options_section )
2016-07-09 20:28:03 +08:00
add_diagnostic ( " Command is %r " % CMDNAME )
2012-04-04 09:38:25 +08:00
if options_matched == None :
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
print ( " Unable to find options section " , file = sys . stderr )
2012-04-04 09:38:25 +08:00
return False
2019-05-05 18:09:25 +08:00
while options_matched != None :
2012-04-04 09:38:25 +08:00
data = options_matched . group ( 1 )
2013-07-06 23:27:41 +08:00
data = remove_groff_formatting ( data )
2012-04-04 09:38:25 +08:00
data = data . strip ( )
2019-05-05 18:09:25 +08:00
data = data . split ( " \n " , 1 )
2012-04-04 09:38:25 +08:00
2019-05-05 18:09:25 +08:00
if len ( data ) > 1 : # and len(data[1])<400):
2012-04-04 09:38:25 +08:00
optionName = data [ 0 ] . strip ( )
2019-05-05 18:09:25 +08:00
if optionName . find ( " - " ) == - 1 :
2016-07-09 20:28:03 +08:00
add_diagnostic ( " %r doesn ' t contain ' - ' " % optionName )
2012-04-04 09:38:25 +08:00
else :
2013-07-06 23:27:41 +08:00
optionName = unquote_double_quotes ( optionName )
optionName = unquote_single_quotes ( optionName )
2019-05-05 18:09:25 +08:00
optionDescription = data [ 1 ] . strip ( ) . replace ( " \n " , " " )
2013-02-18 08:14:36 +08:00
built_command ( optionName , optionDescription )
2012-04-04 09:38:25 +08:00
else :
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Unable to split option from description " )
2012-04-04 09:38:25 +08:00
return False
2019-05-05 18:09:25 +08:00
options_section = options_section [ options_matched . end ( ) - 3 : ]
2012-04-04 09:38:25 +08:00
options_matched = re . search ( options_parts_regex , options_section )
2020-11-22 21:39:48 +08:00
2020-07-09 18:20:09 +08:00
class TypeScdocManParser ( ManParser ) :
def is_my_type ( self , manpage ) :
2020-11-22 21:39:48 +08:00
return (
compile_and_search (
r " \ .( \\ )( \" ) Generated by scdoc(.*?) \ .SH OPTIONS(.*?) " , manpage
)
!= None
)
2020-07-09 18:20:09 +08:00
def parse_man_page ( self , manpage ) :
options_section_regex = re . compile ( " \ .SH OPTIONS(.*?) \ .SH " , re . DOTALL )
options_section_matched = re . search ( options_section_regex , manpage )
if options_section_matched == None :
return False
options_section = options_section_matched . group ( 1 )
options_parts_regex = re . compile ( " (.*?).RE " , re . DOTALL )
options_matched = re . match ( options_parts_regex , options_section )
add_diagnostic ( " Command is %r " % CMDNAME )
if options_matched == None :
add_diagnostic ( " %r : Unable to find options " % self )
return False
while options_matched != None :
# Get first option and move options_section
option = options_matched . group ( 1 )
2020-11-22 21:39:48 +08:00
options_section = options_section [ options_matched . end ( ) : : ]
2020-07-09 18:20:09 +08:00
options_matched = re . match ( options_parts_regex , options_section )
option = remove_groff_formatting ( option )
option = option . split ( " \n " )
# Crean option list
option_clean = [ ]
for line in option :
if line not in [ " " , " .P " , " .RS 4 " ] :
option_clean . append ( line )
# Shold be at least two lines
if len ( option_clean ) < 2 :
add_diagnostic ( " Unable to split option from description " )
continue
2020-11-22 21:39:48 +08:00
2020-07-09 18:20:09 +08:00
# Name and description, others lines are ignored
option_name = option_clean [ 0 ]
option_description = option_clean [ 1 ]
if " - " not in option_name :
add_diagnostic ( " %r doesn ' t contain ' - ' " % option_name )
option_name = unquote_double_quotes ( option_name )
option_name = unquote_single_quotes ( option_name )
built_command ( option_name , option_description )
2012-04-04 09:38:25 +08:00
return True
2019-05-05 18:09:25 +08:00
2012-04-08 16:43:30 +08:00
class TypeDarwinManParser ( ManParser ) :
2013-07-06 23:27:41 +08:00
def is_my_type ( self , manpage ) :
2020-07-09 18:20:09 +08:00
return compile_and_search ( " \ .S[hH] DESCRIPTION " , manpage ) != None
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
def trim_groff ( self , line ) :
# Remove initial period
2019-05-05 18:09:25 +08:00
if line . startswith ( " . " ) :
2012-04-04 09:38:25 +08:00
line = line [ 1 : ]
# Skip leading groff crud
2019-05-05 18:09:25 +08:00
while re . match ( " [A-Z][a-z] \ s " , line ) :
2012-04-04 09:38:25 +08:00
line = line [ 3 : ]
2016-10-10 05:11:04 +08:00
2012-11-23 12:32:19 +08:00
# If the line ends with a space and then a period or comma, then erase the space
# This hack handles lines of the form '.Ar projectname .'
2019-05-05 18:09:25 +08:00
if line . endswith ( " , " ) or line . endswith ( " . " ) :
2012-11-23 12:32:19 +08:00
line = line [ : - 2 ] + line [ - 1 ]
2012-04-04 09:38:25 +08:00
return line
2012-11-18 18:23:22 +08:00
2012-11-23 12:32:19 +08:00
def count_argument_dashes ( self , line ) :
# Determine how many dashes the line has using the following regex hack
# Look for the start of a line, followed by a dot, then a sequence of
# one or more dashes ('Fl')
result = 0
2019-05-05 18:09:25 +08:00
if line . startswith ( " . " ) :
2012-11-23 12:32:19 +08:00
line = line [ 4 : ]
2019-05-05 18:09:25 +08:00
while line . startswith ( " Fl " ) :
2012-11-23 12:32:19 +08:00
result = result + 1
line = line [ 3 : ]
return result
2012-10-18 09:22:57 +08:00
# Replace some groff escapes. There's a lot we don't bother to handle.
def groff_replace_escapes ( self , line ) :
2019-05-05 18:09:25 +08:00
line = line . replace ( " .Nm " , CMDNAME )
line = line . replace ( " \\ " , " " )
line = line . replace ( " \ & " , " " )
2012-10-18 09:22:57 +08:00
return line
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
def is_option ( self , line ) :
2019-05-05 18:09:25 +08:00
return line . startswith ( " .It Fl " )
2012-11-18 18:23:22 +08:00
2013-07-06 23:27:41 +08:00
def parse_man_page ( self , manpage ) :
2012-04-05 03:43:12 +08:00
got_something = False
2019-05-05 18:09:25 +08:00
lines = manpage . splitlines ( )
2012-04-04 09:38:25 +08:00
# Discard lines until we get to ".sh Description"
2019-05-05 18:09:25 +08:00
while lines and not (
lines [ 0 ] . startswith ( " .Sh DESCRIPTION " )
or lines [ 0 ] . startswith ( " .SH DESCRIPTION " )
) :
2012-04-04 09:38:25 +08:00
lines . pop ( 0 )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
while lines :
# Pop until we get to the next option
while lines and not self . is_option ( lines [ 0 ] ) :
lines . pop ( 0 )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
if not lines :
continue
2012-11-18 18:23:22 +08:00
2012-10-18 09:22:57 +08:00
# Get the line and clean it up
line = lines . pop ( 0 )
2016-10-10 05:11:04 +08:00
2012-11-23 12:32:19 +08:00
# Try to guess how many dashes this argument has
dash_count = self . count_argument_dashes ( line )
2016-10-10 05:11:04 +08:00
2012-10-18 09:22:57 +08:00
line = self . groff_replace_escapes ( line )
line = self . trim_groff ( line )
line = line . strip ( )
2019-05-05 18:09:25 +08:00
if not line :
continue
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
# Extract the name
2012-10-18 09:22:57 +08:00
name = line . split ( None , 2 ) [ 0 ]
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
# Extract the description
2012-11-23 12:32:19 +08:00
desc_lines = [ ]
2012-04-04 09:38:25 +08:00
while lines and not self . is_option ( lines [ 0 ] ) :
2013-09-19 04:32:41 +08:00
line = lossy_unicode ( lines . pop ( 0 ) . strip ( ) )
2017-07-21 08:43:46 +08:00
# Ignore comments
2019-05-05 18:09:25 +08:00
if line . startswith ( r " . \" " ) :
2017-07-21 08:43:46 +08:00
continue
2019-05-05 18:09:25 +08:00
if line . startswith ( " . " ) :
2012-11-23 12:32:19 +08:00
line = self . groff_replace_escapes ( line )
line = self . trim_groff ( line ) . strip ( )
if line :
desc_lines . append ( line )
2019-05-05 18:09:25 +08:00
desc = " " . join ( desc_lines )
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
if name == " - " :
2012-04-04 09:38:25 +08:00
# Skip double -- arguments
continue
elif len ( name ) > 1 :
# Output the command
2019-05-05 18:09:25 +08:00
built_command ( ( " - " * dash_count ) + name , desc )
2012-04-05 03:43:12 +08:00
got_something = True
2012-04-04 09:38:25 +08:00
elif len ( name ) == 1 :
2019-05-05 18:09:25 +08:00
built_command ( " - " + name , desc )
2012-04-05 03:43:12 +08:00
got_something = True
2012-11-18 18:23:22 +08:00
2012-04-05 03:43:12 +08:00
return got_something
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
class TypeDeroffManParser ( ManParser ) :
2013-07-06 23:27:41 +08:00
def is_my_type ( self , manpage ) :
2019-05-05 18:09:25 +08:00
return True # We're optimists
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
def is_option ( self , line ) :
2019-05-05 18:09:25 +08:00
return line . startswith ( " - " )
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
def could_be_description ( self , line ) :
2019-05-05 18:09:25 +08:00
return len ( line ) > 0 and not line . startswith ( " - " )
2012-11-18 18:23:22 +08:00
2013-07-06 23:27:41 +08:00
def parse_man_page ( self , manpage ) :
2012-04-08 16:43:30 +08:00
d = Deroffer ( )
d . deroff ( manpage )
output = d . get_output ( )
2019-05-05 18:09:25 +08:00
lines = output . split ( " \n " )
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
got_something = False
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
# Discard lines until we get to DESCRIPTION or OPTIONS
2019-05-05 18:09:25 +08:00
while lines and not (
lines [ 0 ] . startswith ( " DESCRIPTION " )
or lines [ 0 ] . startswith ( " OPTIONS " )
or lines [ 0 ] . startswith ( " COMMAND OPTIONS " )
) :
2012-04-08 16:43:30 +08:00
lines . pop ( 0 )
2012-11-18 18:23:22 +08:00
2012-04-09 14:26:26 +08:00
# Look for BUGS and stop there
2012-06-05 19:05:53 +08:00
for idx in range ( len ( lines ) ) :
2012-04-09 14:26:26 +08:00
line = lines [ idx ]
2019-05-05 18:09:25 +08:00
if line . startswith ( " BUGS " ) :
2012-04-09 14:26:26 +08:00
# Drop remaining elements
lines [ idx : ] = [ ]
break
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
while lines :
# Pop until we get to the next option
while lines and not self . is_option ( lines [ 0 ] ) :
2012-04-09 14:26:26 +08:00
line = lines . pop ( 0 )
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
if not lines :
continue
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
options = lines . pop ( 0 )
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
# Pop until we get to either an empty line or a line starting with -
2019-05-05 18:09:25 +08:00
description = " "
2012-04-08 16:43:30 +08:00
while lines and self . could_be_description ( lines [ 0 ] ) :
2019-05-05 18:09:25 +08:00
if description :
description + = " "
2012-04-08 16:43:30 +08:00
description + = lines . pop ( 0 )
2012-11-18 18:23:22 +08:00
2013-02-18 08:14:36 +08:00
built_command ( options , description )
2012-04-08 16:43:30 +08:00
got_something = True
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
return got_something
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
2012-06-19 04:59:07 +08:00
# Return whether the file at the given path is overwritable
# Raises IOError if it cannot be opened
def file_is_overwritable ( path ) :
result = False
2016-05-30 04:13:13 +08:00
file = codecs . open ( path , " r " , encoding = " utf-8 " )
2012-06-19 04:59:07 +08:00
for line in file :
# Skip leading empty lines
line = line . strip ( )
if not line :
continue
2012-11-18 18:23:22 +08:00
2012-06-19 04:59:07 +08:00
# We look in the initial run of lines that start with #
2019-05-05 18:09:25 +08:00
if not line . startswith ( " # " ) :
2012-06-19 04:59:07 +08:00
break
2012-11-18 18:23:22 +08:00
2012-06-19 04:59:07 +08:00
# See if this contains the magic word
2019-05-05 18:09:25 +08:00
if " Autogenerated " in line :
2012-06-19 04:59:07 +08:00
result = True
break
2012-11-18 18:23:22 +08:00
2012-06-19 04:59:07 +08:00
file . close ( )
return result
2016-10-10 05:11:04 +08:00
2019-05-05 18:09:25 +08:00
2013-02-18 08:14:36 +08:00
# Remove any and all autogenerated completions in the given directory
def cleanup_autogenerated_completions_in_directory ( dir ) :
2012-04-12 09:26:26 +08:00
try :
2013-02-18 08:14:36 +08:00
for filename in os . listdir ( dir ) :
# Skip non .fish files
2019-05-05 18:09:25 +08:00
if not filename . endswith ( " .fish " ) :
continue
2013-02-18 08:14:36 +08:00
path = os . path . join ( dir , filename )
2015-11-08 01:33:07 +08:00
cleanup_autogenerated_file ( path )
2013-02-18 08:14:36 +08:00
except OSError as err :
return False
2012-11-18 18:23:22 +08:00
2012-06-19 04:59:07 +08:00
# Delete the file if it is autogenerated
def cleanup_autogenerated_file ( path ) :
try :
if file_is_overwritable ( path ) :
os . remove ( path )
except ( OSError , IOError ) :
pass
2012-04-12 09:26:26 +08:00
2020-11-22 21:39:48 +08:00
2013-02-18 08:14:36 +08:00
def parse_manpage_at_path ( manpage_path , output_directory ) :
2020-07-09 04:43:38 +08:00
# Return if CMDNAME is in 'ignoredcommands'
ignoredcommands = [
" cc " ,
" g++ " ,
" gcc " ,
" c++ " ,
" cpp " ,
" emacs " ,
" gprof " ,
" wget " ,
" ld " ,
" awk " ,
]
if CMDNAME in ignoredcommands :
return
2012-04-04 09:38:25 +08:00
# Clear diagnostics
global diagnostic_indent
diagnostic_output [ : ] = [ ]
diagnostic_indent = 0
# Set up some diagnostics
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Considering " + manpage_path )
2012-04-04 09:38:25 +08:00
diagnostic_indent + = 1
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
if manpage_path . endswith ( " .gz " ) :
fd = gzip . open ( manpage_path , " r " )
2012-06-05 19:22:29 +08:00
manpage = fd . read ( )
2019-05-05 18:09:25 +08:00
if IS_PY3 :
manpage = manpage . decode ( " latin-1 " )
elif manpage_path . endswith ( " .bz2 " ) :
fd = bz2 . BZ2File ( manpage_path , " r " )
2013-09-24 21:42:32 +08:00
manpage = fd . read ( )
2019-05-05 18:09:25 +08:00
if IS_PY3 :
manpage = manpage . decode ( " latin-1 " )
elif manpage_path . endswith ( " .xz " ) or manpage_path . endswith ( " .lzma " ) :
2013-09-25 20:36:42 +08:00
if not lzma_available :
return
2019-05-05 18:09:25 +08:00
fd = lzma . LZMAFile ( str ( manpage_path ) , " r " )
2013-09-24 21:42:32 +08:00
manpage = fd . read ( )
2019-05-05 18:09:25 +08:00
if IS_PY3 :
manpage = manpage . decode ( " latin-1 " )
2015-11-08 01:12:03 +08:00
elif manpage_path . endswith ( ( " .1 " , " .2 " , " .3 " , " .4 " , " .5 " , " .6 " , " .7 " , " .8 " , " .9 " ) ) :
2012-06-19 04:59:07 +08:00
if IS_PY3 :
2019-05-05 18:09:25 +08:00
fd = open ( manpage_path , " r " , encoding = " latin-1 " )
2012-06-19 04:59:07 +08:00
else :
2019-05-05 18:09:25 +08:00
fd = open ( manpage_path , " r " )
2012-06-05 19:05:53 +08:00
manpage = fd . read ( )
2015-11-08 01:12:03 +08:00
else :
return
2012-04-04 09:38:25 +08:00
fd . close ( )
2012-06-05 19:22:29 +08:00
2012-06-05 19:05:53 +08:00
manpage = str ( manpage )
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
# Ignore perl's gazillion man pages
2019-05-05 18:09:25 +08:00
ignored_prefixes = [ " perl " , " zsh " ]
2012-04-08 16:43:30 +08:00
for prefix in ignored_prefixes :
2020-07-09 04:43:38 +08:00
if CMDNAME . startswith ( prefix ) :
2012-04-08 16:43:30 +08:00
return
2012-11-18 18:23:22 +08:00
2012-04-08 16:43:30 +08:00
# Ignore the millions of links to BUILTIN(1)
2019-05-05 18:09:25 +08:00
if " BUILTIN 1 " in manpage or " builtin.1 " in manpage :
2012-04-08 16:43:30 +08:00
return
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
# Clear the output list
built_command_output [ : ] = [ ]
2012-11-18 18:23:22 +08:00
2012-06-19 04:59:07 +08:00
if DEROFF_ONLY :
parsers = [ TypeDeroffManParser ( ) ]
else :
2019-05-05 18:09:25 +08:00
parsers = [
2020-07-09 18:20:09 +08:00
TypeScdocManParser ( ) ,
2019-05-05 18:09:25 +08:00
Type1ManParser ( ) ,
Type2ManParser ( ) ,
Type4ManParser ( ) ,
Type3ManParser ( ) ,
TypeDarwinManParser ( ) ,
TypeDeroffManParser ( ) ,
]
2013-07-06 23:27:41 +08:00
parsersToTry = [ p for p in parsers if p . is_my_type ( manpage ) ]
2012-11-18 18:23:22 +08:00
2012-04-05 03:43:12 +08:00
success = False
2020-07-09 18:20:09 +08:00
for parser in parsersToTry :
add_diagnostic ( " Trying %s " % parser . __class__ . __name__ )
diagnostic_indent + = 1
success = parser . parse_man_page ( manpage )
diagnostic_indent - = 1
# Make sure empty files aren't reported as success
if not built_command_output :
success = False
2012-04-04 09:38:25 +08:00
if success :
2020-07-09 18:20:09 +08:00
PARSER_INFO . setdefault ( parser . __class__ . __name__ , [ ] ) . append ( CMDNAME )
break
if success :
if WRITE_TO_STDOUT :
output_file = sys . stdout
2012-04-04 09:38:25 +08:00
else :
2020-07-09 18:20:09 +08:00
fullpath = os . path . join ( output_directory , CMDNAME + " .fish " )
try :
output_file = codecs . open ( fullpath , " w " , encoding = " utf-8 " )
except IOError as err :
add_diagnostic (
2020-11-22 21:39:48 +08:00
" Unable to open file ' %s ' : error( %d ): %s "
2020-07-09 18:20:09 +08:00
% ( fullpath , err . errno , err . strerror )
)
return False
# Output the magic word Autogenerated so we can tell if we can overwrite this
2020-11-22 21:39:48 +08:00
built_command_output . insert (
0 , " # " + CMDNAME + " \n # Autogenerated from man page " + manpage_path
)
2020-07-09 18:20:09 +08:00
# built_command_output.insert(2, "# using " + parser.__class__.__name__) # XXX MISATTRIBUTES THE CULPABLE PARSER! Was really using Type2 but reporting TypeDeroffManParser
for line in built_command_output :
output_file . write ( line )
output_file . write ( " \n " )
output_file . write ( " \n " )
add_diagnostic ( manpage_path + " parsed successfully " )
if output_file != sys . stdout :
output_file . close ( )
else :
parser_names = " , " . join ( p . __class__ . __name__ for p in parsersToTry )
# add_diagnostic('%s contains no options or is unparsable' % manpage_path, BRIEF_VERBOSE)
add_diagnostic (
" %s contains no options or is unparsable (tried parser %s ) "
% ( manpage_path , parser_names ) ,
BRIEF_VERBOSE ,
)
2015-11-08 01:25:16 +08:00
2012-04-05 03:43:12 +08:00
return success
2012-01-29 14:25:45 +08:00
2019-05-05 18:09:25 +08:00
2013-02-18 08:14:36 +08:00
def parse_and_output_man_pages ( paths , output_directory , show_progress ) :
2012-06-19 04:59:07 +08:00
global diagnostic_indent , CMDNAME
2012-06-05 19:05:53 +08:00
paths . sort ( )
2012-04-05 03:43:12 +08:00
total_count = len ( paths )
2012-04-12 10:05:43 +08:00
successful_count , index = 0 , 0
padding_len = len ( str ( total_count ) )
last_progress_string_length = 0
if show_progress and not WRITE_TO_STDOUT :
2019-05-05 18:09:25 +08:00
print (
" Parsing man pages and writing completions to {0} " . format ( output_directory )
)
2013-09-26 23:15:53 +08:00
man_page_suffixes = set ( [ os . path . splitext ( m ) [ 1 ] [ 1 : ] for m in paths ] )
lzma_xz_occurs = " xz " in man_page_suffixes or " lzma " in man_page_suffixes
if lzma_xz_occurs and not lzma_available :
2019-05-05 18:09:25 +08:00
add_diagnostic (
' At least one man page is compressed with lzma or xz, but the " lzma " module is not available. '
" Any man page compressed with either will be skipped. " ,
NOT_VERBOSE ,
)
2013-09-26 23:15:53 +08:00
flush_diagnostics ( sys . stderr )
2012-04-04 09:38:25 +08:00
for manpage_path in paths :
2012-04-12 10:05:43 +08:00
index + = 1
2012-11-18 18:23:22 +08:00
2012-06-19 04:59:07 +08:00
# Get the "base" command, e.g. gcc.1.gz -> gcc
man_file_name = os . path . basename ( manpage_path )
2019-05-05 18:09:25 +08:00
CMDNAME = man_file_name . split ( " . " , 1 ) [ 0 ]
2012-11-18 18:23:22 +08:00
2012-06-19 04:59:07 +08:00
# Show progress if we're doing that
2012-04-12 10:05:43 +08:00
if show_progress :
2019-05-05 18:09:25 +08:00
progress_str = " {0} / {1} : {2} " . format (
( str ( index ) . rjust ( padding_len ) ) , total_count , man_file_name
)
2012-04-12 10:05:43 +08:00
# Pad on the right with spaces so we overwrite whatever we wrote last time
padded_progress_str = progress_str . ljust ( last_progress_string_length )
last_progress_string_length = len ( progress_str )
2012-07-09 08:35:07 +08:00
sys . stdout . write ( " \r {0} \r " . format ( padded_progress_str ) )
2012-06-19 04:59:07 +08:00
sys . stdout . flush ( )
2012-11-18 18:23:22 +08:00
2012-04-04 09:38:25 +08:00
try :
2013-02-18 08:14:36 +08:00
if parse_manpage_at_path ( manpage_path , output_directory ) :
2012-04-05 03:43:12 +08:00
successful_count + = 1
2012-04-04 09:38:25 +08:00
except IOError :
diagnostic_indent = 0
2019-05-05 18:09:25 +08:00
add_diagnostic ( " Cannot open " + manpage_path )
2012-04-08 16:43:30 +08:00
except ( KeyboardInterrupt , SystemExit ) :
raise
2012-04-04 09:38:25 +08:00
except :
2019-05-05 18:09:25 +08:00
add_diagnostic (
" Error parsing %s : %s " % ( manpage_path , sys . exc_info ( ) [ 0 ] ) ,
BRIEF_VERBOSE ,
)
2012-04-04 09:38:25 +08:00
flush_diagnostics ( sys . stderr )
traceback . print_exc ( file = sys . stderr )
flush_diagnostics ( sys . stderr )
2019-05-05 18:09:25 +08:00
print ( " " ) # Newline after loop
add_diagnostic (
" Successfully parsed %d / %d pages " % ( successful_count , total_count ) ,
BRIEF_VERBOSE ,
)
2012-04-05 03:43:12 +08:00
flush_diagnostics ( sys . stderr )
2012-04-12 09:26:26 +08:00
2019-05-05 18:09:25 +08:00
2017-12-11 22:08:02 +08:00
def get_paths_from_man_locations ( ) :
2013-05-23 22:47:07 +08:00
# Return all the paths to man(1) and man(8) files in the manpath
2019-05-05 18:09:25 +08:00
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
parent_paths = [ ]
# Most (GNU, macOS, Haiku) modern implementations of man support being called with `--path`.
# Traditional implementations require a second `manpath` program: examples include FreeBSD and Solaris.
# Prefer an external program first because these programs return a superset of the $MANPATH variable.
for prog in [ [ " man " , " --path " ] , [ " manpath " ] ] :
try :
output = subprocess . check_output ( prog , stderr = DEVNULL )
if IS_PY3 :
output = output . decode ( " latin-1 " )
parent_paths = output . strip ( ) . split ( " : " )
2020-05-10 21:56:12 +08:00
break
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
except ( OSError , subprocess . CalledProcessError ) :
continue
# If we can't have the OS interpret $MANPATH, just use it as-is (gulp).
if not parent_paths and os . getenv ( " MANPATH " ) :
2019-05-05 18:09:25 +08:00
parent_paths = os . getenv ( " MANPATH " ) . strip ( ) . split ( " : " )
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
# Fallback: With mandoc (OpenBSD, embedded Linux) and NetBSD man, the only way to get the default manpath is by reading /etc.
if not parent_paths :
try :
with open ( " /etc/man.conf " , " r " ) as file :
data = file . read ( )
for key in [ " MANPATH " , " _default " ] :
for match in re . findall ( r " ^ %s \ s+(.*)$ " % key , data , re . I | re . M ) :
parent_paths . append ( match )
except FileNotFoundError :
pass
# Fallback: hard-code some common paths. These should be likely for FHS Linux distros, BSDs, and macOS.
if not parent_paths :
parent_paths = [ " /usr/share/man " , " /usr/local/man " , " /usr/local/share/man " ]
print (
" Unable to get the manpath, falling back to %s . " % " : " . join ( parent_paths ) ,
" Explictly set $MANPATH to fix this error. " ,
2020-06-25 02:43:56 +08:00
file = sys . stderr ,
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
)
2012-04-12 09:26:26 +08:00
result = [ ]
for parent_path in parent_paths :
2019-05-05 18:09:25 +08:00
for section in [ " man1 " , " man6 " , " man8 " ] :
2013-05-23 22:47:07 +08:00
directory_path = os . path . join ( parent_path , section )
try :
names = os . listdir ( directory_path )
Fix manpath handling in create_manpage_completions.py (#6879)
* Fix manpath handling in create_manpage_completions.py
...as well as do some (very!) light cleanup.
Currently, `create_manpage_completions.py` does not properly
understand/respect the `$MANPATH` variable. One important feature of
`$MANPATH` is that an empty component (i.e. the trailing : in
`foo:bar:`) expands to the 'default' or 'system' path -- that is to say,
the path that would be used if `$MANPATH` was unset. This allows the
user to extend the manpath without clobbering it, and has been a feature
many Unices have included for years.
The current implementation blindly uses the `$MANPATH` variable if it
exists, which does not allow for this behaviour -- to expand the
variable correctly, an external program must be invoked. Therefore, we
first shell out to the 'proper' (read: best guess) external program. If
that fails, we can then try to use `$MANPATH` directly/literally.
Finally, if both of those are impossible, we can fall back to some
common paths from widely used operating systems.
Note that the `man.conf` parsing has been removed: this is because while
many 'traditional' Unices (BSDs, Solaris, macOS) support this file, only
macOS actually ships a file -- most other Unices use a `conf.d`-style
layout and supporting that from our Python is impractical and silly at
best. On GNU (read: Linux) systems, `mandb` uses `/etc/man_db.conf` with
slightly different syntax and sematics. As this code-path has bitrotted
(and likely never worked, anyway), just remove it.
`create_manpage_completions.py` looks like it has suffered a lot of
confusion and bitrot in general over the last few years -- and is
overdue for a major refactoring. I am quite interested in tackling this,
but I plan to wait until the go-ahead to drop support for Python 2 is
given, as a major refactor/rewrite that still supports Python 2 (and
thus ignores the ergonomic/API/syntax improvements of Python 3) does not
make sense to me.
Related: #5657
It would probably be good to revisit `man.fish` once again when a
comprehensive refactor happens: hopefully every permutation of
`man`/`$MANPATH` could be documented as part of that effort.
* Restore /etc/man.conf parsing
I was not aware that this codepath was used -- since it appeared that it
would throw an error when it was reached. Redo it, using regex, and
support parsing NetBSD man.conf as well (untested).
* Fix create_manpage_completions.py under Python 2
2020-05-08 03:01:02 +08:00
except OSError :
2013-05-23 22:47:07 +08:00
names = [ ]
names . sort ( )
for name in names :
result . append ( os . path . join ( directory_path , name ) )
2012-04-12 09:26:26 +08:00
return result
2012-11-18 18:23:22 +08:00
2019-05-05 18:09:25 +08:00
2012-03-31 01:00:01 +08:00
if __name__ == " __main__ " :
2012-04-12 09:26:26 +08:00
script_name = sys . argv [ 0 ]
2020-04-04 20:52:11 +08:00
parser = argparse . ArgumentParser (
description = " create_manpage_completions: Generate fish-shell completions from manpages "
)
parser . add_argument (
" -c " ,
" --cleanup-in " ,
type = str ,
help = " Directories to clean up in " ,
action = " append " ,
)
parser . add_argument (
2020-11-22 21:39:48 +08:00
" -d " ,
" --directory " ,
type = str ,
help = " The directory to save the completions in " ,
2020-04-04 20:52:11 +08:00
)
parser . add_argument (
" -k " ,
" --keep " ,
help = " Whether to keep files in the target directory " ,
action = " store_true " ,
)
parser . add_argument (
2020-11-22 21:39:48 +08:00
" -m " ,
" --manpath " ,
help = " Whether to use manpath " ,
action = " store_true " ,
2020-04-04 20:52:11 +08:00
)
parser . add_argument (
2020-11-22 21:39:48 +08:00
" -p " ,
" --progress " ,
help = " Whether to show progress " ,
action = " store_true " ,
2020-04-04 20:52:11 +08:00
)
parser . add_argument (
2020-11-22 21:39:48 +08:00
" -s " ,
" --stdout " ,
help = " Write the completions to stdout " ,
action = " store_true " ,
2020-04-04 20:52:11 +08:00
)
parser . add_argument (
" -v " ,
" --verbose " ,
type = int ,
choices = [ 0 , 1 , 2 ] ,
help = " The level of debug output to show " ,
)
parser . add_argument (
2020-11-22 21:39:48 +08:00
" -z " ,
" --deroff-only " ,
help = " Whether to just deroff " ,
action = " store_true " ,
2020-04-04 20:52:11 +08:00
)
2020-04-05 02:23:35 +08:00
parser . add_argument ( " file_paths " , type = str , nargs = " * " )
2020-04-04 20:52:11 +08:00
args = parser . parse_args ( )
if args . verbose :
VERBOSITY = args . verbose
if args . stdout :
WRITE_TO_STDOUT = True
if args . deroff_only :
DEROFF_ONLY = True
if args . keep :
KEEP_FILES = True
if args . manpath :
# Fetch all man1 and man8 files from the manpath or man.conf
args . file_paths . extend ( get_paths_from_man_locations ( ) )
2016-10-10 05:11:04 +08:00
2013-02-18 08:14:36 +08:00
# Directories within which we will clean up autogenerated completions
# This script originally wrote completions into ~/.config/fish/completions
# Now it writes them into a separate directory
2020-04-04 20:52:11 +08:00
if args . cleanup_in :
for cleanup_dir in args . cleanup_in :
2013-02-18 08:14:36 +08:00
cleanup_autogenerated_completions_in_directory ( cleanup_dir )
2020-04-04 20:52:11 +08:00
if not args . file_paths :
2012-06-05 19:05:53 +08:00
print ( " No paths specified " )
2012-04-12 09:26:26 +08:00
sys . exit ( 0 )
2016-10-10 05:11:04 +08:00
2020-04-04 20:52:11 +08:00
if not args . stdout and not args . directory :
2015-09-25 21:19:27 +08:00
# Default to ~/.local/share/fish/generated_completions/
2012-04-12 09:26:26 +08:00
# Create it if it doesn't exist
2019-05-05 18:09:25 +08:00
xdg_data_home = os . getenv ( " XDG_DATA_HOME " , " ~/.local/share " )
2020-04-04 20:52:11 +08:00
args . directory = os . path . expanduser (
2019-05-05 18:09:25 +08:00
xdg_data_home + " /fish/generated_completions/ "
)
2012-04-12 09:26:26 +08:00
try :
2020-04-04 20:52:11 +08:00
os . makedirs ( args . directory )
2012-06-05 19:05:53 +08:00
except OSError as e :
2012-04-12 09:26:26 +08:00
if e . errno != errno . EEXIST :
raise
2012-11-18 18:23:22 +08:00
2020-04-04 20:52:11 +08:00
if not args . stdout and not args . keep :
2015-11-08 01:25:16 +08:00
# Remove old generated files
2020-04-04 20:52:11 +08:00
cleanup_autogenerated_completions_in_directory ( args . directory )
2015-11-08 01:25:16 +08:00
2020-04-04 20:52:11 +08:00
parse_and_output_man_pages ( args . file_paths , args . directory , args . progress )