The relevant standards allow the mbtowc/mbrtowc functions to reject
non-ASCII characters (i.e., chars with the high bit set) when the locale
is C or POSIX. The BSD libraries (e.g., on OS X) don't do this but
the GNU libraries (e.g., on Linux) do. Like most programs we need the
C/POSIX locales to allow arbitrary bytes. So explicitly check if we're
in a single-byte locale (which would also include ISO-8859 variants)
and simply pass-thru the chars without encoding or decoding.
Fixes#2802.
This was used to cache a narrow string representation
of commands, so that if certain system calls returned errors
after fork, we could output error messages without allocating
memory. But in practice these errors are very uncommon, as are
commands that have wide characters. It is simpler to do a best-effort
output of the wide string, instead of caching a narrow string
unconditionally.
This change eliminates global variables like stdout_buffer. Instead we wrap up
the IO information into a new struct io_streams_t, and thread that through
every builtin. This makes the intent clearer, gives us a place to hang new IO
data, and eliminates the ugly global state management like builtin_push_io.
This adds the new builtin 'string' which supports various string
manipulation and matching algorithms, including PCRE based regular
expressions.
Fixes#2296
Squashed commit of the following:
commit 4c3eaeb6e57d76463e9683c327142b0aeafb92b8
Author: ridiculousfish <corydoras@ridiculousfish.com>
Date: Sat Sep 12 12:51:30 2015 -0700
Remove testdata and doc dirs from pcre2 source
commit b2a8b4b50f2398b204fb72cfe4b5ba77ece2e1ab
Merge: 11c8a477974aab
Author: ridiculousfish <corydoras@ridiculousfish.com>
Date: Sat Sep 12 12:32:40 2015 -0700
Merge branch 'string' of git://github.com/msteed/fish-shell into string-test
commit 7974aab6d3
Author: Michael Steed <msteed@saltstack.com>
Date: Fri Sep 11 13:00:02 2015 -0600
build pcre2 lib only, no docs
commit eb20b43d2d
Merge: 1a09e705f519cb
Author: Michael Steed <msteed68@gmail.com>
Date: Thu Sep 10 20:00:47 2015 -0600
Merge branch 'string' of github.com:msteed/fish-shell into string
commit 1a09e709d0
Author: Michael Steed <msteed68@gmail.com>
Date: Thu Sep 10 19:58:24 2015 -0600
rebase on master & address the fallout
commit a0ec9772cd
Author: Michael Steed <msteed68@gmail.com>
Date: Thu Sep 10 19:26:45 2015 -0600
use fish's wildcard_match() for glob matching
commit 64c25a01e3
Author: Michael Steed <msteed68@gmail.com>
Date: Thu Aug 27 08:19:23 2015 -0600
some fixes from review
- string_get_arg_stdin(): simplify and don't discard the argument when
the trailing newline is absent
- fix calls to pcre2 for e.g. string match -r -a 'a*' 'b'
- correct test for args coming from stdin
commit ece7f35ec5
Author: Michael Steed <msteed68@gmail.com>
Date: Sat Aug 22 19:35:56 2015 -0600
fixes from review
- Makefile.in: restore iwyu target
- regex_replacer_t::replace_matches(): correct size passed to realloc()
commit 9ff7477a92
Author: Michael Steed <msteed68@gmail.com>
Date: Thu Aug 20 13:08:33 2015 -0600
Minor doc improvements
commit baf4e096b2
Author: Michael Steed <msteed68@gmail.com>
Date: Wed Aug 19 18:29:02 2015 -0600
another attempt to fix the ci build
commit 896a2c2b27
Author: Michael Steed <msteed68@gmail.com>
Date: Wed Aug 19 18:03:49 2015 -0600
Updates after review comments
- make match/replace without -a operate on the first match on each
argument
- use different exit codes for "no operation performed" and errors, as
grep does
- refactor regex compile code
- use human-friendly error messages from pcre2
- improve error handling & reporting elsewhere
- add a few tests
- make some doc fixes
- some simplification & cleanup
- fix ci build failure (I hope)
commit efd47dcbda
Author: Michael Steed <msteed68@gmail.com>
Date: Wed Aug 12 00:26:07 2015 -0600
fix dependencies for parallel make
commit ed0850e2db
Author: Michael Steed <msteed68@gmail.com>
Date: Tue Aug 11 23:37:22 2015 -0600
Add missing pcre2 files + .gitignore
commit 9492e7a7e9
Author: Michael Steed <msteed68@gmail.com>
Date: Tue Aug 11 22:44:05 2015 -0600
add pcre2-10.20 and update license.hdr
commit 1a60b93371
Author: Michael Steed <msteed68@gmail.com>
Date: Tue Aug 11 22:41:19 2015 -0600
add string builtin files
- string builtin source, tests, & docs
- changes to configure.ac & Makefile.in
commit 5f519cb2a2
Author: Michael Steed <msteed68@gmail.com>
Date: Thu Sep 10 19:26:45 2015 -0600
use fish's wildcard_match() for glob matching
commit 2ecd24f795
Author: Michael Steed <msteed68@gmail.com>
Date: Thu Aug 27 08:19:23 2015 -0600
some fixes from review
- string_get_arg_stdin(): simplify and don't discard the argument when
the trailing newline is absent
- fix calls to pcre2 for e.g. string match -r -a 'a*' 'b'
- correct test for args coming from stdin
commit 45b777e4dc
Author: Michael Steed <msteed68@gmail.com>
Date: Sat Aug 22 19:35:56 2015 -0600
fixes from review
- Makefile.in: restore iwyu target
- regex_replacer_t::replace_matches(): correct size passed to realloc()
commit 981cbb6ddf
Author: Michael Steed <msteed68@gmail.com>
Date: Thu Aug 20 13:08:33 2015 -0600
Minor doc improvements
commit ddb6a2a8fd
Author: Michael Steed <msteed68@gmail.com>
Date: Wed Aug 19 18:29:02 2015 -0600
another attempt to fix the ci build
commit 1e34e3191b
Author: Michael Steed <msteed68@gmail.com>
Date: Wed Aug 19 18:03:49 2015 -0600
Updates after review comments
- make match/replace without -a operate on the first match on each
argument
- use different exit codes for "no operation performed" and errors, as
grep does
- refactor regex compile code
- use human-friendly error messages from pcre2
- improve error handling & reporting elsewhere
- add a few tests
- make some doc fixes
- some simplification & cleanup
- fix ci build failure (I hope)
commit 34232e152d
Author: Michael Steed <msteed68@gmail.com>
Date: Wed Aug 12 00:26:07 2015 -0600
fix dependencies for parallel make
commit 00d7e78169
Author: Michael Steed <msteed68@gmail.com>
Date: Tue Aug 11 23:37:22 2015 -0600
Add missing pcre2 files + .gitignore
commit 4498aa5f57
Author: Michael Steed <msteed68@gmail.com>
Date: Tue Aug 11 22:44:05 2015 -0600
add pcre2-10.20 and update license.hdr
commit 290c58c72e
Author: Michael Steed <msteed68@gmail.com>
Date: Tue Aug 11 22:41:19 2015 -0600
add string builtin files
- string builtin source, tests, & docs
- changes to configure.ac & Makefile.in
Rather than trying to detect Unicode support from the environment, check
the printable width of characters in the current locale before deciding
on whether to use them.
Closes#1927.
This change moves source files into a src/ directory,
and puts object files into an obj/ directory. The Makefile
and xcode project are updated accordingly.
Fixes#1866
