Commit Graph

58 Commits

Author SHA1 Message Date
ridiculousfish
1a4b1c3298 Remove the is_first parameter from tok_is_string_character
This parameter is unused now that carets are no longer special, per
7f905b082.
2022-04-16 10:47:01 -07:00
ridiculousfish
1763e7d3bc Remove some dead code
These functions were unused.
2022-03-20 14:48:44 -07:00
Shay Aviv
2ef12af60e Fix comment parsing inside command substitutions and brackets 2022-02-08 16:20:31 +01:00
Aaron Gyes
365a6ee384 Use source_offset_t (uint32) in tokenizer.
Seems like size_t is unnecessarily large as well, as elsewhere
in the code we are clamping down to uint32_t / source_offset_t.

This makes tok_t more like 16 bytes. More cleanup seems desirable,
this is not very well hamrnoized across our code base.
2021-12-21 02:32:52 -08:00
Aaron Gyes
35253900cb rearrange tok_t fields in decreasing order
We can have vectors of these, maybe it adds up.
40 bytes -> 32 bytes
2021-12-21 01:29:26 -08:00
Aaron Gyes
db72a05e89 Tighten up includes, some typedefs -> using
clangd was unable to resolve some symbols
2021-09-21 18:05:53 -07:00
ridiculousfish
5f7e03ccf4 Introduce noncopyable_t and nonmovable_t
These are little helper types that allow us to get rid of lots of
'=delete' declarations.
2021-07-23 11:19:42 -07:00
ridiculousfish
cc07716dc1 Separate out variable assignments when completing
In preparation for applying variable assignments (VAR=VAL cmd), separate
them out from the command when performing completions. This includes both
those that the user typed, and any that come about through
completion --wraps.
2020-09-26 17:30:25 -07:00
ridiculousfish
72e35af381 Remove preceding_escaped_nl
It's no longer necessary for fish_indent
2020-07-07 13:48:35 -07:00
Johannes Altmanninger
aa0e16b1a5 Skip variable assignments in status current command
Fixes #6635
2020-02-24 00:16:12 +01:00
Johannes Altmanninger
ebde9a6a44 move variable_assignment_equal_pos to tokenizer
we'll need it for tok_command
2020-02-24 00:14:39 +01:00
ridiculousfish
af473d4d0c Introduce redirection_spec_t
Prior to this change, a process after it has been constructed by
parse_execution, but before it is executed, was given a list of
io_data_t redirections. The problem is that redirections have a
sensitive ownership policy because they hold onto fds. This made it
rather hard to reason about fd lifetime.

Change these to redirection_spec_t. This is a textual description
of a redirection after expansion. It does not represent an open file and
so its lifetime is no longer important.

This enables files to be held only on the stack, and are no longer owned
by a process of indeterminate lifetime.
2019-12-12 16:44:24 -08:00
ridiculousfish
eaa87ff885 Correct a comment
A comment claimed that 0 was STDOUT_FILENO. In fact this should be 1.

Also default the fd for pipe_or_redir_t to -1, as 0 is not a sensible
default.
2019-12-10 16:14:34 -08:00
Rosen Penev
5ca80a61e3 [clang-tidy] Fix inconsistent declarations
Found with readability-inconsistent-declaration-parameter-name

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2019-11-25 14:13:33 -08:00
Jason
3cf6ebc0e1 Amend typos and grammar errors 2019-11-25 13:07:15 +01:00
ridiculousfish
896ef65f8c Rename error_offset to error_offset_within_token
Hopefully clarify the role of this variable.
2019-11-08 16:56:50 -08:00
ridiculousfish
eb2386f3e3 Rename tokenizer_t::buff to token_cursor
That should clarify its role vis-a-vis 'start'.
2019-11-08 16:56:18 -08:00
Johannes Altmanninger
6fb7f9b6b8 Fix completion for builtins (with subcommands)
Presently the completion engine ignores builtins that are part of the
fish syntax. This can be a problem when completing a string that was
based on the output of `commandline -p`.  This changes completions to
treat these builtins like any other command.

This also disables generic (filename) completion inside comments and
after strings that do not tokenize.

Additionally, comments are stripped off the output of `commandline -p`.

Fixes #5415
Fixes #2705
2019-11-04 16:44:51 +01:00
ridiculousfish
eeac3333df Correctly highlight input following a tokenizer error 2019-10-27 16:08:49 -07:00
ridiculousfish
afd20b8e1a Correctly report the range of tokenizer errors
This enables proper syntax highlighting of tokenizer errors.
2019-10-27 16:05:37 -07:00
ridiculousfish
0bfd897ee0 Add a special error message for |& 2019-10-27 15:24:57 -07:00
ridiculousfish
2a92e66902 Support for &> and &| as convenience redirections
This adds support for &> and &| syntax, which both redirect stdout, and
also apply a redirection of stderr to stdout.
2019-10-27 15:24:57 -07:00
ridiculousfish
35671dd9f0 Clean up and unify pipes and redirections
This cleans up how pipes and redirections are recognized by the parser,
and unifies pipes and redirections into a single type.
2019-10-15 11:26:41 -07:00
ridiculousfish
4b6de8b4a3 Clean up a stale comment. 2019-10-13 23:05:55 -07:00
ridiculousfish
c160492d42 Remove tok_t::redirected_fd
This wasn't used for anything.
2019-10-13 23:04:18 -07:00
ridiculousfish
1a65e18ba8 Clean up some tokenization
Remove TOK_NONE
Turn token_type into an enum class
Make next() turn a maybe_t<tok_t> instead of a bool
2019-10-13 16:06:16 -07:00
Aaron Gyes
d837eee09d remove some wcstring -> wchar_t* -> wcstring conversions
Mostly related to usage _(L"foo"), keeping in mind the _
macro does a wcstring().c_str() already.

And a smattering of other trivial micro-optimizations certain
to not help tangibly.
2019-03-14 15:21:08 -07:00
ridiculousfish
cc99e8d510 Switch tokenizer_error back to just an error code
Rather than having tokenizer_error as pointers to objects, switch it back
to just an error code value. This makes reasoning about it easier since
it's immutable values instead of mutable objects, and it avoids allocation
during startup.
2018-09-27 21:40:51 -04:00
ridiculousfish
6f57fef8f8 Teach the tokenizer to report escaped newlines
Add fields and flags so that escaped newlines can be reported, for the
benefit of fish_indent.
2018-05-07 21:39:30 -07:00
Mahmoud Al-Qudsi
1441cca9c5 Restore localization to tokenizer error strings
Work around #4810 by retrieving localizations at runtime to avoid issues
possibly caused by inserting into the static unordered_map during static
initialization.

Closes #810.
2018-03-13 13:45:15 -05:00
Mahmoud Al-Qudsi
0f6c763d9e Revert "Restore sanity to line continuations"
This reverts commit 2cdacbdce4.
Our fish scripts need to be audited for compliance before this can be
safely merged. cc @faho
2018-03-12 08:35:09 -05:00
Mahmoud Al-Qudsi
2cdacbdce4 Restore sanity to line continuations
Line continuations (i.e. escaped new lines) now make sense again. With
the smart pipe support (pipes continue on to next line) recently added,
this hack to have continuations ignore comments makes no sense.

This is valid code:

```fish
echo hello |
# comment here
tr -d 'l'
```

this isn't:
```fish
echo hello | \
# comment here
tr -d 'l'
```

Reverts @snnw's 318daaffb2
Closes #2928. Closes #2929.
2018-03-12 08:12:23 -05:00
Mahmoud Al-Qudsi
00f95a978e Make { and } valid, first-class tokenizer elements 2018-03-11 19:36:10 -05:00
Mahmoud Al-Qudsi
f508a1f274 Reset tokenizer state on start and improve slice error detection 2018-03-11 12:13:55 -05:00
Mahmoud Al-Qudsi
fb1c7a3d68 Simplify token parser 2018-03-10 18:42:56 -06:00
ridiculousfish
8ded041352 Add && and || support to tokenizer 2018-03-05 12:20:56 -08:00
ridiculousfish
c4d903ff98 Rationalize how the parser reports tokenizer errors
Remove the unnecessary SQUASH_ERROR flag and correctly report errors
generated from the tokenizer.
2018-02-23 17:28:12 -08:00
ridiculousfish
0950c35eb2 Reduce the amount of copying when the parser drives the tokenizer 2018-02-23 15:58:13 -08:00
ridiculousfish
99fb7bb6aa Refactor how redirections are represented by the tokenizer
Prior to this fix, each redirection type was a separate token_type.
Unify these under a single type TOK_REDIRECT and break the redirection
type out into a new sub-type redirection_type_t.
2018-02-23 15:19:58 -08:00
ridiculousfish
6673fe5457 Clean up tokenizer implementation
Rather than storing a bunch of "next_foo" fields, simply populate the
tok_t directly.
2018-02-23 14:31:13 -08:00
ridiculousfish
0f62161b2b Clean up tokenizer error handling. 2018-02-19 16:31:39 -08:00
ridiculousfish
f30bf40300 Clean up comment parsing in tokenizer
Unify the show_comments and non-show_comments path.
2018-02-19 16:20:47 -08:00
ridiculousfish
b13ee818d2 Some early cleanup of tokenizer
Prior to this the tokenizer ran "one ahead", where tokenizer_t::next()
would in fact return the last-parsed token. Switch to parsing on demand
instead of running one ahead; this is simpler and prepares for tokenizer
changes.
2018-02-19 15:10:59 -08:00
ridiculousfish
fd2a0dffa9 Reflect newlines after pipes in fish grammar
The previous attempt to support newlines after pipes changed the lexer to
swallow newlines after encountering a pipe. This has two problems that are
difficult to fix:

1. comments cannot be placed after the pipe
2. fish_indent won't know about the newlines, so it will erase them

Address these problems by removing the lexer behavior, and replacing it
with a new parser symbol "optional_newlines" allowing the newlines to be
reflected directly in the fish grammar.
2018-02-18 14:44:08 -08:00
slama
38418d6356 Enable multi-line edit when the line ends with a pipe (#1285) 2018-02-18 13:01:38 -08:00
Aaron Gyes
fa78a7101c Make IWYU output in lint.cpp less messy
And re-run IWYU, adjust #includes.
2016-06-23 17:26:08 -07:00
Kurtis Rader
c14bac4284 restyle tokenizer module to match project style
Reduces lint errors from 70 to 46 (-34%). Line count from 1158 to 936 (-19%).

Another step in resolving issue #2902.
2016-05-03 15:00:44 -07:00
Kurtis Rader
1f06e5f0b9 add better support for IWYU and fix things
Remove the "make iwyu" build target. Move the functionality into the
recently introduced lint.fish script. Fix a lot, but not all, of the
include-what-you-use errors. Specifically, it fixes all of the IWYU errors
on my OS X server but only removes some of them on my Ubuntu 14.04 server.

Fixes #2957
2016-04-26 15:02:22 -07:00
ridiculousfish
10f3ea0008 Mark a bunch of constructors as explicit
This prevents undesired implicit conversions
2016-02-27 19:38:15 -08:00
ridiculousfish
aa76f64a94 Correct the caret position for unbalanced square brackets 2015-08-10 19:30:21 -07:00