Commit Graph

336 Commits

Author SHA1 Message Date
ridiculousfish
64ce1088fd Make function_prepare_environment take argv instead of the process
This will help concurrent execution.
2019-11-14 17:58:39 -08:00
ridiculousfish
b51edcfcac Simplify function_info_t and function_data_t
Work towards cleaning up function definition. Migrate inherit_vars into
props and capture their values at the point of definition.
2019-11-12 09:53:10 -08:00
ridiculousfish
b0cf94e3ba Refactor function_prepare_environment
Migrate it into exec.cpp to reduce the complexity of
exec_block_or_func_process.
2019-11-10 14:46:21 -08:00
ridiculousfish
ac2eed2ffa Make io_chain_t store const io_data_t
This will make it easier to reason about with concurrent execution.
2019-11-10 14:00:30 -08:00
ridiculousfish
521d0e84f5 Remove non-const get_io_for_fd
These could be made unused.
2019-11-10 13:41:12 -08:00
ridiculousfish
2555ecf757 Remove the forbidden function stack
Detect forbidden functions directly from the associated block_t.
Also unify where we do stack overflow detection.
2019-11-10 12:36:46 -08:00
ridiculousfish
a7f1d2c0c7 Add support for fish_trace variable to trace execution
This adds support for `fish_trace`, a new variable intended to serve the
same purpose as `set -x` as in bash. Setting this variable to anything
non-empty causes execution to be traced. In the future we may give more
specific meaning to the value of the variable.

The user's prompt is not traced unless you run it explicitly. Events are
also not traced because it is noisy; however autoloading is.

Fixes #3427
2019-11-02 14:40:57 -07:00
ridiculousfish
cc1c973025 Remove job_flags as an enum, just use a struct
This removes an over-complicated flag implementation, replacing it with
just a plain struct.
2019-10-15 14:40:58 -07:00
ridiculousfish
82eca4bc86 Run clang-format on all files
The main change here is to reorder headers.
2019-10-13 15:50:48 -07:00
Aaron Gyes
22811ebcf6 Remove unused macros
Most just left behind after actual code was moved into different files.
2019-09-18 01:31:03 -07:00
ridiculousfish
73bf60754c Don't assign jobs to fish's pgrp if there's no external process
Fixes #6011
2019-08-03 14:43:55 -07:00
ridiculousfish
a33f0eb636 Clean up some logic around when process exit events are sent 2019-07-28 14:36:57 -07:00
ridiculousfish
1f854ec284 Accept a pwd in resolve_file_redirections_to_fds
Conceptually allow multiple of these to run in parallel
2019-07-20 17:25:25 -07:00
ridiculousfish
8181883111 Minor refactoring of logic around when a job wants to claim the terminal
Introduce should_claim_terminal() which encapsulates an && exprsesion which
was previously repeated a lot.
2019-07-12 13:31:56 -07:00
ridiculousfish
b1a1b617f1 child_setup_process to accept new termowner directly
Soon we will have more complicated logic around whether to call tcsetpgrp.
Prepare to centralize the logic by passing in the new term owner pgrp,
instead of having child_setup_process perform the decision.
2019-07-03 18:06:35 -07:00
ridiculousfish
8282369f45 child_setup_process to stop passing the process
child_setup_process only cares about whether we are in a forked child, not
the entire process structure. Narrow the parameter.
2019-07-03 17:48:52 -07:00
ridiculousfish
09e4f8ff42 Refactor how the terminal is transferred to jobs
Centralize the logic around when a job acquires the terminal.
2019-06-29 15:58:36 -07:00
ridiculousfish
98ba7d7790 Simplify maybe_assign_terminal()
Move this out of postfork, it is not called after fork.
2019-06-29 14:36:14 -07:00
ridiculousfish
f58960ba01 Add a DIE("unreachable") in should_claim_process_group_for_job
Fixes a gcc warning.
2019-06-29 11:40:52 -07:00
ridiculousfish
f3736e8fdf fish to claim a job's pgroup if the first process is fish internal
When executing a job, if the first process is fish internal, then have
fish claim the job's pgroup.

The idea here is that the terminal must be owned by a pgroup containing
the process reading from the terminal. If the first process is fish
internal (a function or builtin) then the pgroup must contain the fish
process.

This is a bit of a workaround of the behavior where the first process that
executes in a job becomes the process group leader. If there's a deferred
process, then we will execute processes out of order so the pgroup can be
wrong. Fix this by setting the process group leader explicitly as fish
when necessary.

Fixes #5855
2019-06-26 17:30:51 -07:00
ridiculousfish
89fb408eb6 Migrate some job flags into const properties struct
This helps clarify which parts of a job are mutable, and which are constant.
2019-06-23 12:42:44 -07:00
ridiculousfish
5362161343 Call tcsetgrp() in child processes again
25afc9b377 made this unnecessary by
having child processes wait for a signal after fork(), but this change
was later reverted. If we artificially slow down fish (e.g. with a sleep)
after the fork call, we see commands getting backgrounded by mistake.

Put back the tcsetgrp() call.
2019-06-23 12:42:44 -07:00
ridiculousfish
8255c01edb Rename setup_child_process to child_setup_process
Try to name all functions called after fork with "child" prefix.
2019-06-23 12:42:44 -07:00
ridiculousfish
6ce85aebc6 Switch file_io_t to store a wcstring
We no longer use file_io_t after fork(). We don't need to use a malloc'd
string any more. Use a wcstring.
2019-06-09 17:43:25 -07:00
ridiculousfish
b478f877ee Clean up g_fork_count
Make it static and atomic
2019-06-03 12:58:59 -07:00
Fabian Homborg
87971e1f2e Widen the rest of the FLOGs
Fixes #5900.
2019-05-30 13:08:35 +02:00
Fabian Homborg
d73ee4d54b More using FLOGF when formatting is needed
sed-patched, every time a "%" is used in a call to `FLOG`, we use
`FLOGF` instead.
2019-05-30 11:54:09 +02:00
ridiculousfish
f3ee6a99c3 Add some FLOG logging around internal processes 2019-05-29 12:34:11 -07:00
Fabian Homborg
32949eefbe src/exec: Use wide IO for FLOG
Another step towards #5900.
2019-05-28 23:08:35 +02:00
ridiculousfish
835c6ffa67 clang-format all files 2019-05-27 19:47:13 -07:00
ridiculousfish
ea9d1ad82f Convert debug(0) calls to FLOG 2019-05-27 17:31:17 -07:00
ridiculousfish
d9676bb2a8 Convert fork events from debug to flog 2019-05-27 17:24:52 -07:00
ridiculousfish
8774860468 Convert job logging from debug to FLOG 2019-05-27 17:24:52 -07:00
ridiculousfish
a5a5ccb73d Revert "exec: If a job includes a builtin, it stays in our pgroup"
This reverts commit 711260593c.

This ended up breaking the tests; moreover it may have other problems
like preventing backgrounding.

Reopens #5855
2019-05-24 18:00:31 -07:00
Fabian Homborg
711260593c exec: If a job includes a builtin, it stays in our pgroup
Fixes #5855.
2019-05-24 18:04:20 +02:00
ridiculousfish
4d929720ce Clean up and rename io_transmorgrify 2019-05-22 16:36:22 -07:00
ridiculousfish
e91d68266c Eliminate reader_current_filename
Store this in the parser libdata instead.
2019-05-22 13:51:27 -07:00
ridiculousfish
686b84396c Migrate the return bool outside of block_t
This is a flag that gets set by the return function. But we only need one,
not per-block. Move it into libdata.
2019-05-22 13:51:27 -07:00
ridiculousfish
eff4873eca Stop creating subclasses of block_t
Move all block_t creation methods to static methods, and stop creating
subclasses (all of which are now empty).
2019-05-19 14:40:35 -07:00
ridiculousfish
c42eb0eb4f Remove the process from function_block_t
Prior to this fix, a function_block stored a process_t, which was only used
when printing backtraces. Switch this to an array of arguments, and make
various other cleanups around null terminated argument arrays.
2019-05-18 21:09:04 -07:00
ridiculousfish
508c3a8005 Make is_event and other globals part of parser_t libdata 2019-05-18 19:03:45 -07:00
ridiculousfish
4fcb9d1fed Hide no_exec behind a function 2019-05-18 18:50:28 -07:00
ridiculousfish
1719d6f136 Make $status and $pipestatus per-parser
Another step towards allowing multiple parsers to execute in parallel.
2019-05-12 14:00:44 -07:00
ridiculousfish
16fd780484 Reimplement the whole variable stack
The variable stack is a mess - confused locking, surprising callouts, and
unclear division of labor. Just reimplement the whole thing.
2019-05-11 19:17:16 -07:00
ridiculousfish
8a8b2513b5 Eliminate the global jobs() function
All job lists are attached to a parser now.
2019-05-05 11:33:08 -07:00
Fabian Homborg
c2970f9618 Reformat all files
This runs build_tools/style.fish, which runs clang-format on C++, fish_indent on fish and (new) black on python.

If anything is wrong with the formatting, we should fix the tools, but automated formatting is worth it.
2019-05-05 12:09:25 +02:00
ridiculousfish
9fb98baba6 Thread the parser into process_clean_after_marking 2019-05-04 20:58:35 -07:00
ridiculousfish
b0a695e9fa Migrate the global exec_count into the parser 2019-05-04 20:58:35 -07:00
ridiculousfish
acd33cbabb Remove an unnecessary fetch of the principal parser 2019-05-04 16:56:38 -07:00
Mahmoud Al-Qudsi
8e4010b263 Make eval override previous status
Closes #5692 (again).
2019-04-13 17:28:10 -05:00
Mahmoud Al-Qudsi
4530a41004 Fix return code after execution of empty function 2019-04-13 17:21:13 -05:00
ridiculousfish
a597b0e6e1 Remove get_proc_had_barrier
Prior to this change, fish used a global flag to decide if we should check
for changes to universal variables. This flag was then checked at arbitrary
locations, potentially triggering variable updates and event handlers for
those updates; this was very hard to reason about.

Switch to triggering a universal variable update at a fixed location,
after running an external command.  The common case is that the variable
file has not changed, which we can identify with just a stat() call, so
this is pretty cheap.
2019-04-13 12:40:57 -07:00
ridiculousfish
13c5f93d63 Revert "Optimize function calls by reducing inherit vars heap allocations and copies"
This reverts commit cdce8511a1.

This change was unsafe. The prior version (now restored) took the lock and
then copied the data. By returning a reference, the caller holds a
reference to data outside of the lock.

This function isn't worth optimizing. Hardly any functions use this
facility, and for those that do, they typically just capture one or two
variables.
2019-04-13 12:03:02 -07:00
Mahmoud Al-Qudsi
cdce8511a1 Optimize function calls by reducing inherit vars heap allocations and copies
* Convert `function_get_inherit_vars()` to return a reference to the
  (possibly) existing map, rather than a copy;
* Preallocate and reuse a static (read-only) map for the (very) common
  case of no inherited vars;
* Pass references to the inherit vars map around thereafter, never
  triggering the map copy (or even move) constructor.

NB: If it turns out the reference is unsafe, we can switch the inherit vars
to be a shared_ptr and return that instead.
2019-04-13 11:26:10 -05:00
ridiculousfish
47b9907113 Remove an unused variable 2019-04-12 23:03:28 -07:00
Mahmoud Al-Qudsi
e0e0fe9dd3 Re-implement eval as a regular builtin
I did not realize builtins could safely call into the parser and inject
jobs during execution. This is much cleaner than hacking around the
required shape of a plain_statement.
2019-04-12 07:04:15 -05:00
Mahmoud Al-Qudsi
4539a9db15 Drop unused include in src/exec.cpp
It was added in 2544c622841fd8b7317109f12fe4eb55c5ea1d0a,
and caught by @faho.
2019-04-11 13:01:29 -05:00
Mahmoud Al-Qudsi
2fe2169065 Make eval a decorator
`eval` has always been implemented as a function, which was always a bit
of a hack that caused some issues such as triggering the creation of a
new scope. This turns `eval` into a decorator.

The scoping issues with eval prevented it from being usable to actually
implement other shell components in fish script, such as the problems
described in #4442, which should now no longer be the case.

Closes #4443.
2019-04-11 10:36:49 -05:00
Mahmoud Al-Qudsi
0bda853dc7 Add detection of eval to the parser
While `eval` is still a function, this paves the way for changing that
in the future, and lets the proc/exec functions detect when an eval is
used to allow/disallow certain behaviors and optimizations.
2019-04-10 21:19:57 -05:00
ridiculousfish
f5bb8639d6 More aggressively inherit pgrps from parent jobs
Prior to this fix, a job would only inherit a pgrp from its parent if the
first command were external. There seems to be no reason for this
restriction and this causes tcsetgrp() churn, potentially cuasing SIGTTIN.
Switch to unconditionally inheriting a pgrp from parents.

This should fix most of #5765, the only remaining question is
tcsetpgrp from builtins.
2019-04-07 13:35:00 -07:00
ridiculousfish
39a9740997 Be less aggressive about reclaiming the foreground pgrp
Prior to this fix, in every call to job_continue, fish would reclaim the
foreground pgrp. This would cause other jobs in the pipeline (which may
have another pgrp) to receive SIGTTIN / SIGTTOU.

Only reclaim the foreground pgrp if it was held at the point of job_continue.

This partially addresses #5765
2019-04-07 09:20:32 -07:00
Mahmoud Al-Qudsi
3c537bfa65 Optimize get_deferred_process() traversal 2019-03-31 13:20:49 -05:00
ridiculousfish
0b11b8cffb Revert "Optimize identification of deferred process"
This reverts commit 4aea4c09b3.

Said commit broke many tests
2019-03-28 20:22:58 -07:00
Mahmoud Al-Qudsi
4aea4c09b3 Optimize identification of deferred process 2019-03-28 22:12:50 -05:00
ridiculousfish
5eade35257 Stop buffering deferred function processes
If a function process is deferred, allow it to be unbuffered.
This permits certain simple cases where functions are piped to external
commands to execute without buffering.

This is a somewhat-hacky stopgap measure that can't really be extended
to more general concurrent processes. However it is overall an improvement
in user experience that might help flush out some bugs too.
2019-03-24 21:23:26 -07:00
ridiculousfish
3bbee06248 Introduce the notion of a deferred process
In a job, a deferred process is the last fish internal process which pipes
to an external command. Execute the deferred process last; this will allow
for streaming its output.
2019-03-24 14:27:23 -07:00
ridiculousfish
165c82e68a Promote process_type_t to an enum class 2019-03-24 12:29:25 -07:00
Aaron Gyes
477b2e8d7c std::vector<wcstring> is wcstring_list_t 2019-03-14 11:17:26 -07:00
Aaron Gyes
aaacdb89b6 Switches over to cstring from string.h. 2019-03-12 15:09:36 -07:00
ridiculousfish
1a4bb50cd5 Combine status and pipestatus into statuses_t
In most places where we set one, we want to set both. Make this less
error-prone by combining them into a single type statuses_t.
2019-02-26 20:07:37 -08:00
ridiculousfish
bb36274e6b Introduce proc_status_t
In fish we play fast and loose with status codes as set directly (e.g. on
failed redirections), vs status codes returned from waitpid(), versus the
value $status. Introduce a new value type proc_status_t to encapsulate
this logic.
2019-02-25 10:14:45 -08:00
ridiculousfish
2c3214cabd Make $pipestatus thread safe and other misc cleanup 2019-02-24 23:29:33 -08:00
zabereer
2c8abdf5cb add $pipestatus support 2019-02-24 21:46:52 -08:00
ridiculousfish
0b3eca1743 Cleanup handle_builtin_output
Now that we use an internal process to perform builtin output, simplify the
logic around how it is performed. In particular we no longer have to be
careful about async-safe functions since we do not fork.

Also fix a bunch of comments that no longer apply.
2019-02-17 14:17:44 -08:00
ridiculousfish
4a2fd443b2 Use internal processes to write builtin output
This uses the new internal process mechanism to write output for builtins.
After this the only reason fish ever forks is to execute external processes.
2019-02-17 13:08:00 -08:00
ridiculousfish
ada8ea954e Use "internal" processes to write buffered output
This introduces "internal processes" which are backed by a pthread instead
of a normal process. Internal processes are reaped using the topic
machinery, plugging in neatly alongside the sigchld topic; this means that
process_mark_finished_children() can wait for internal and external
processes simultaneously.

Initially internal processes replace the forked process that fish uses to
write out the output of blocks and functions.
2019-02-17 13:05:20 -08:00
ridiculousfish
ebe2dc2766 Processes to record topic generations before execution
The sigchld generation expresses the idea that, if we receive a sigchld
signal, the generation will be different than when we last recorded it. A
process cannot exit before it has launched, so check the generation count
before process launch. This is an optimization that reduces failing
waitpid calls.
2019-02-17 13:01:59 -08:00
Mrmaxmeier
6e9250425a src/exec: fix assertion on failed exec redirection
Minimal reproducer: `fish -c "exec cat<x"`
2019-02-12 20:52:03 -08:00
ridiculousfish
1701e2c558 Revert "add $pipestatus support"
This reverts commit ec290209db.
2019-02-10 13:46:58 -08:00
zabereer
ec290209db add $pipestatus support 2019-02-10 13:30:40 -08:00
David Adam
662708e72d src/exec: drop unused parameter in can_use_posix_spawn_for_job
Process object is not checked since 084ff64f4f.
2019-02-10 15:57:06 +08:00
ridiculousfish
d3fa58d621 Cleanup common.h
Remove a bunch of headers, simplify lots of code, migrate it into .cpp files.

Debug build time improves by ~3 seconds on my Mac.
2019-02-03 18:22:38 -08:00
ridiculousfish
6f682c8405 Fill io_buffer via background thread
This is a large change to how io_buffers are filled. The essential problem
comes about with code like (example):

    echo ( /bin/pwd )

The output of /bin/pwd must go to fish, not the tty. To arrange for this,
fish does the following:

1. Invoke pipe() to create a pipe.
2. Add an io_bufferfill_t redirection that owns the write end of the pipe.
3. After fork (or equiv), call dup2() to replace pwd's stdout with this  pipe.

Now when /bin/pwd writes, it will send output to the read end of the pipe.
But who reads it?

Prior to this fix, fish would do the following in a loop:

1. select() on the pipe with a 10 msec timeout
2. waitpid(WNOHANG) on the pwd proc

This polling is ugly and confusing and is what is replaced here.

With this new change, fish now reads from the pipe via a background thread:

1. Spawn a background pthread, which select()s on the pipe's read end with
a long (100 msec) timeout.
2. In the foreground, waitpid() (allowing hanging) on the pwd proc.

The big win here is a major simplification of job_t::continue_job() since
it no longer has to worry about filling buffers. This will make things
easier for concurrent execution.

It may not be obvious why the background thread still needs a poll (100 msec).
The answer is for cases where the write end of the fd escapes, in particular
background processes invoked inside command substitutions. psub is perhaps
the only important case of this (other shells typically just hang here).
2019-02-03 01:58:49 -08:00
ridiculousfish
178b72b2fd io_buffer_t becomes io_bufferfill_t
This makes some significant architectual improvements to io_pipe_t and
io_buffer_t.

Prior to this fix, io_buffer_t subclassed io_pipe_t. io_buffer_t is now
replaced with a class io_bufferfill_t, which does not subclass pipe.

io_pipe_t no longer remembers both fds. Instead it has an autoclose_fd_t,
so that the file descriptor ownership is clear.
2019-02-03 01:58:49 -08:00
ridiculousfish
084ff64f4f Allow posix_spawn more often
Now that we no longer open files after fork, we can correctly report errors
for failed file opens. So allow posix_spawn even if there's redirections.
2019-02-03 01:58:49 -08:00
ridiculousfish
2742267b9e Use dup2_list_t in posix_spawn
This simplifies the posix_spawn path and unifies it with the fork execution
path.
2019-02-03 01:58:49 -08:00
ridiculousfish
d62576ce22 Adopt dup2_list_t in fork execution path
This switches IO redirections after fork() to use the dup2_list_t,
instead of io_chain_t. This results in simpler code with much simpler
error handling.
2019-02-03 01:58:49 -08:00
ridiculousfish
6ba0d4c88a Revert io_bufferfill_t stack
This reverts commit 88dc484858 onwards.
2019-02-02 17:53:40 -08:00
ridiculousfish
9a4153f5e2 Fill io_buffer via background thread
This is a large change to how io_buffers are filled. The essential problem
comes about with code like (example):

    echo ( /bin/pwd )

The output of /bin/pwd must go to fish, not the tty. To arrange for this,
fish does the following:

1. Invoke pipe() to create a pipe.
2. Add an io_bufferfill_t redirection that owns the write end of the pipe.
3. After fork (or equiv), call dup2() to replace pwd's stdout with this  pipe.

Now when /bin/pwd writes, it will send output to the read end of the pipe.
But who reads it?

Prior to this fix, fish would do the following in a loop:

1. select() on the pipe with a 10 msec timeout
2. waitpid(WNOHANG) on the pwd proc

This polling is ugly and confusing and is what is replaced here.

With this new change, fish now reads from the pipe via a background thread:

1. Spawn a background pthread, which select()s on the pipe's read end with
a long (100 msec) timeout.
2. In the foreground, waitpid() (allowing hanging) on the pwd proc.

The big win here is a major simplification of job_t::continue_job() since
it no longer has to worry about filling buffers. This will make things
easier for concurrent execution.

It may not be obvious why the background thread still needs a poll (100 msec).
The answer is for cases where the write end of the fd escapes, in particular
background processes invoked inside command substitutions. psub is perhaps
the only important case of this (other shells typically just hang here).
2019-02-02 14:21:46 -08:00
ridiculousfish
78bbcef356 io_buffer_t becomes io_bufferfill_t
This makes some significant architectual improvements to io_pipe_t and
io_buffer_t.

Prior to this fix, io_buffer_t subclassed io_pipe_t. io_buffer_t is now
replaced with a class io_bufferfill_t, which does not subclass pipe.

io_pipe_t no longer remembers both fds. Instead it has an autoclose_fd_t,
so that the file descriptor ownership is clear.
2019-02-02 14:21:46 -08:00
ridiculousfish
7c256e7e51 Allow posix_spawn more often
Now that we no longer open files after fork, we can correctly report errors
for failed file opens. So allow posix_spawn even if there's redirections.
2019-02-02 14:21:46 -08:00
ridiculousfish
4c0b6a6add Use dup2_list_t in posix_spawn
This simplifies the posix_spawn path and unifies it with the fork execution
path.
2019-02-02 14:21:46 -08:00
ridiculousfish
d895075d9b Adopt dup2_list_t in fork execution path
This switches IO redirections after fork() to use the dup2_list_t,
instead of io_chain_t. This results in simpler code with much simpler
error handling.
2019-02-02 14:21:46 -08:00
ridiculousfish
b00f039489 Clean up the io_chain_t interface 2019-01-31 18:49:52 -08:00
ridiculousfish
371f67f1b5 Remove pipe_read_fd
In practice it was always STDIN_FILENO.
2019-01-31 17:58:59 -08:00
ridiculousfish
a2aab24db7 Switch io_mode to an enum class 2019-01-31 12:12:46 -08:00
ridiculousfish
6f52e6bb1c Instantize contents of exec.cpp and others 2019-01-10 20:07:47 -08:00
ridiculousfish
c1dd284b3e Instantize env_set
Switch env_set to an instance method on environmnet_t.
2019-01-10 20:05:45 -08:00
ridiculousfish
ede66ccaac Instance env_set_argv and env_set_pwd 2019-01-10 20:29:10 -08:00
ridiculousfish
e6872b83b0 Eliminate global env_export_arr()
This assumes the set of exported variables is a global property; but we
want it to be a local property.
2019-01-10 20:29:10 -08:00
Mahmoud Al-Qudsi
3d557518d5 Replace 0/1 with true/false in calls to job_reap 2018-11-18 17:40:18 -06:00
Mahmoud Al-Qudsi
d0085cae3c Fix zombie job on failed redirection in exec_job
Closes #5346.
2018-11-18 17:40:18 -06:00
ridiculousfish
73537fc7c3 Remove NESTED and WAIT_BY_PROCESS
Now jobs are aware of their parent jobs, and can interrogate those jobs,
to determine if every job in the chain is fully constructed.
Remove flags and the static stacks that manipulated them.
2018-11-04 01:52:17 -08:00
ridiculousfish
3770d9fb7a Teach each job about its parent
The parent of a job is the parent pipeline that executed the function or
block corresponding to this job. This will help simplify
process_mark_finished_children().
2018-11-04 01:40:07 -08:00
ridiculousfish
93aa95d8c4 Remove proc_last_bg_pid
It wasn't used.
2018-11-03 19:28:16 -07:00
Mahmoud Al-Qudsi
4d3b56c151 Associate external commands in functions with extant pgrps
When a function is encountered by exec_job, a new context is created for
its execution from the ground up, with a new job and all, ultimately
resulting in a recursive call to exec_job from the same (main) thread.

Since each time exec_job encounters a new job with external commands
that needs terminal control it creates a new pgrp and gives it control
of the terminal (tcsetpgrp & co), this effectively takes control away
from the previously spawned external commands which may be (and likely
are) expecting to still have terminal access.

This commit attempts to detect when such a situation arises by handling
recursive calls to exec_job (which can only happen if the pipeline
included a function) by borrowing the pgrp from the (necessarily still
active) parent job and spawning new external commands into it.

When a parent job spawns new jobs due to the evaluation of a new
function (which shouldn't be the case in the first place), we end up
with two distinct jobs sharing one pgrp (to fix #3952). This can lead to
early termination of a pgrp if finished parent job children are reaped
before future processes in either the parent or future child jobs can
join it.

While the parent job is under construction, require that waitpid(2)
calls for the child job be done by process id and not job pgrp.

Closes #3952.
2018-10-27 18:01:38 -05:00
Mahmoud Al-Qudsi
419d7a5138 Don't decompose shared_ptr to raw pointer for exec_job 2018-10-27 18:01:38 -05:00
Mahmoud Al-Qudsi
3afcca3114 Drop keepalive process even for WSL
Windows 10 17763 Redstone 5 (October 2018 Update) officially brings
zombie support (first introduced in 17713) to the general public.

See https://docs.microsoft.com/en-us/windows/wsl/release-notes#build-17763-1809
2018-10-27 18:01:38 -05:00
Mahmoud Al-Qudsi
f9118d964e Clean up job flags, status helpers, and instance helper methods
* Convert JOB_* enums to scoped enums
* Convert standalone job_is_* functions to member functions
* Convert standalone job_{promote, signal, continue} to member functions
* Convert standolen job_get{,_from_pid} to `job_t` static functions
* Reduce usage of JOB_* enums outside of proc.cpp by using new
  `job_t::is_foo()` const helper methods instead.

This patch is only a refactor and should not change any functionality or
behavior (both observed and unobserved).
2018-10-27 18:01:38 -05:00
Mahmoud Al-Qudsi
e753581df7 Bring some consistency and rationale to debug log levels
* Debug level 3: describe all commands being executed (this is, after all,
a shell and one can argue that this is the most important debug
information avaliable)
* Debug level 4: details of execution, mainly fork vs no-fork and io
handling

Also introduced j->preview() to print a short descriptor of the job
based on the head of the first process so we don't overwhelm with
needless repitition, but also so that we don't have to rely on
distinguishing between repeated, non-unique/non-monotonic job ids that
are often recycled within a single "execution cycle" (pressing enter
once).
2018-10-27 18:01:38 -05:00
Mahmoud Al-Qudsi
d467bb58d9 Replace pid/pgid -2 with INVALID_PID 2018-10-27 18:01:38 -05:00
Mahmoud Al-Qudsi
af0c8d51e0 Overhaul job and terminal control
* Instead of reaping all child processes when we receive a SIGCHLD, try
reaping only processes belonging to process groups from fully-
constructed jobs, which should eliminate the need for the keepalive
process entirely (WSL's lack of zombies not withstanding) as now
completed processes are not reaped until the job has been fully
constructed (i.e.  all processes launched), which means their process
group should still be around for new processes to join.

* When `tcgetpgrp()` calls return 0, attempt to `tcsetpgrp()` before
invoking failure handling code.

* When forking a builtin and not running interactively, do not bail if
unable to set/restore terminal attributes.

Fixes #4178. Fixes #3805. Fixes #5210.
2018-10-27 18:01:38 -05:00
ridiculousfish
a17a815c87 Revert "Add vector of cleanup/termination events to be executed before quit"
This reverts commit 8c14f0f30f.

This list is not reliable - there are many ways for fish to quit that does not
invoke these functions. It's also not necessary since the history is correctly
saved on exec.
2018-09-28 20:21:23 -04:00
Mahmoud Al-Qudsi
8c14f0f30f Add vector of cleanup/termination events to be executed before quit 2018-09-28 11:34:07 -05:00
ridiculousfish
ca61fc1bf8 Stop retrying close() on EINTR
https://lwn.net/Articles/576478/
http://austingroupbugs.net/view.php?id=529
https://sourceware.org/bugzilla/show_bug.cgi?id=14627
2018-09-05 21:49:31 -07:00
ridiculousfish
8b277e711e Large refactor of exec.cpp
Break up that monster function.
2018-09-03 15:57:11 -07:00
ridiculousfish
eca4d113c6 Factor do_fork into a real function 2018-09-03 14:33:53 -07:00
ridiculousfish
2a62e18635 Remove child_forked and child_spawned
These variables weren't used for anything.
2018-09-03 13:31:03 -07:00
ridiculousfish
f7a020ad33 Rename launch process to exec_process_in_job
This avoids a name collision with another launch_process
2018-09-03 11:18:39 -07:00
ridiculousfish
48c510572b Factor out launch_process from exec.cpp
Makes the monster function slightly more tractable.
2018-09-01 14:54:23 -07:00
ridiculousfish
753639aa9c Reduce the scope of pid in exec_job 2018-09-01 14:39:32 -07:00
ridiculousfish
ec9c592edc Adopt autoclose_fd_t in exec_job 2018-09-01 14:27:58 -07:00
ridiculousfish
d9f34147c3 builtins to only acquire terminal if owned by their pgroup
Fix #5133 changed builtins to acquire the terminal, but this regressed
caused fish to be stopped when running in background via `sudo fish`.
Fix this by only acquiring the terminal if the terminal was owned by the
builtin's pgroup.

Fixes #5147
2018-08-18 16:56:01 -07:00
ridiculousfish
d34a300818 Add string split0
This adds a new string command split0, which splits on zero bytes.
split0 has superpowers because its output is not further split on
newlines when used in command substitutions.
2018-07-01 15:56:33 -07:00
ridiculousfish
f998afaa23 Adopt separated_buffer_t in io_buffer_t 2018-07-01 15:56:33 -07:00
ridiculousfish
90a4af5112 Add separated_buffer_t and adopt it in output_stream_t
separated_buffer_t encapsulates the logic around discarding (which
was previously duplicated between output_stream_t and io_buffer_t),
and will also encapsulate the logic around explicitly separated
output.
2018-07-01 15:56:33 -07:00
ridiculousfish
5b9331ade0 Teach io_buffer_t to append from output_stream_t directly
This will simplify logic when we teach output_stream_t about explicitly
split outputs, i.e. for 'string split0'
2018-07-01 15:56:33 -07:00
ridiculousfish
2443ea92c3 Eliminate a common subexpression 2018-06-16 11:43:52 -07:00
Mahmoud Al-Qudsi
0dd2607cac Iron out situation with setpgid() calls after posix_spawn()
Closes #4715. Ticks off a box in #4154.
2018-05-16 19:34:56 -05:00
Aaron Miller
517b77ca74 Fix handling of signals (#4851) 2018-03-24 12:37:15 -07:00
Mahmoud Al-Qudsi
26cc112096 Implement $last_pid, taking the place of %last
Set as a global variable upon the execution of a background job.
2018-03-09 08:56:13 -06:00
Mahmoud Al-Qudsi
2f2a221c56 Don't spawn keepalive for WSL when only one command
This should speed things up on slower PCs given that the vast majority
of shell commands are simple jobs consisting of a single command without
any pipelines, in which case there's no need for a keepalive process at
all. Applies to WSL only.
2018-03-04 21:54:12 -06:00
Mahmoud Al-Qudsi
cf8850a33f Add temporary fix for #4778 (background processes on WSL)
As a temporary workaround for the behavior described in
Microsoft/WSL#2997 wherein WSL does not correctly assign the spawned
child its own PID as its PGID, explicitly set the PGID for the newly
spawned process.
2018-03-04 20:18:38 -06:00
Mahmoud Al-Qudsi
50541544f2 Distinguish between function and block IO for fork debug log messages 2018-02-18 16:49:27 -06:00
Mahmoud Al-Qudsi
be13ac353b Refactor job control to make functions act like their names imply
The job control functions were a bit messy, in particular
`set_child_group`'s name would imply that all it does is set the child
group, but in reality it used to set the child group (via `setpgid`),
set the job's pgrp if it hasn't been set, and possibly assign control of
the terminal to the newly-created job.

These have been split into separate functions. Now `set_child_group`
does just (and only) that, `maybe_assign_terminal` might assign the
terminal to the new pgrp, and `on_process_created` is used to set the
job properties the first time an external process is created. This might
also speed things up (but probably not noticeably) as there are no more
repeated calls to `getpgrp()` if JOB_CONTROL is not set.

Additionally, this closes #4715 by no longer unconditionally calling
`setpgid` on all new processes, including those created by `posix_spawn`
which does not need this since the child's pgrep is set at in the
arguments to that API call.
2018-02-14 19:08:12 -06:00
ridiculousfish
c3f1961e36 Stop copying out function definition when executing a function
This switches function execution from the function's source code to
its stored node and pstree. This means we no longer have to re-parse
the function every time we execute it.
2018-02-12 10:55:00 -08:00
ridiculousfish
976514597d Migrate function getters to use function_get_properties
This replaces some of the teensy function getters with the function
that just returns a shared_ptr to the properties struct.
2018-02-12 10:53:22 -08:00
ridiculousfish
41ba0dfadb Evaluate tnode_t instead of parse_node_t
This concerns block nodes with redirections, like
begin ... end | grep ...
Prior to this fix, we passed in a pointer to the node. Switch to passing
in the tnode and parsed source ref. This improves type safety and better
aligns with the function-node plans.
2018-02-12 10:51:39 -08:00
ridiculousfish
cb03be9fe6 Remove unused 'pgrp_set' variable 2018-02-07 12:54:26 -08:00
ridiculousfish
8de266afb4 Improve commenting regarding process groups and builtins. 2018-02-07 12:49:12 -08:00
ridiculousfish
0c18f68cc2 Remove support for blocking children
This removes support for blocking children via signals, which was used
to orchestrate processes on WSL. Now we use the keepalive mechanism
instead.
2018-02-07 12:49:12 -08:00
ridiculousfish
080521071f Teach keepalives to exit when their parent dies
keepalive processes are typically killed by the main shell process.
However if the main shell exits the keepalive may linger. In WSL
keepalives are used more often, and the lingering keepalives are both
leaks and prevent the tests from finishing.

Have keepalives poll for their parent process ID and exit when it
changes, so they can clean themselves up. The polling frequency can be
low.
2018-02-07 12:49:12 -08:00
ridiculousfish
e9f676a7f4 Provide a way to stop blocking children via s_block_children
This is to investigate alternatives to the existing kill(SIGSTOP)
WSL compatibility thing.
2018-02-07 12:49:11 -08:00
ridiculousfish
1b1fd5ab9b Mark needs_keepalive more often for WSL
Have WSL use a keepalive whenever the first process is external.
This works around the fact that WSL prohibits setting an exited
process as the group leader.
2018-02-07 12:49:11 -08:00
ridiculousfish
d17b298a48 Factor out the code that executes a builtin from exec_job()
Very early work on untangling the exec_job spaghetti.
2017-12-22 13:41:29 -08:00
Aaron Gyes
a9283803d4 Revert "Non-exported vars: rename SHLVL to shlvl"
Duh, of course it is exported.

This reverts commit 5fc17dcc82.
2017-10-15 04:37:34 -07:00
Aaron Gyes
5fc17dcc82 Non-exported vars: rename SHLVL to shlvl
Fixes #4414
2017-10-15 04:33:27 -07:00
Mahmoud Al-Qudsi
9fdfe44236 Fix type of pid_status variable
We had pid_status defined as a pid_t instance, which was fine since on
most platforms pid_t is an alias for int. However, that is not
universally the case and waitpid takes an int *, not a pid_t *.
2017-09-26 08:16:36 -05:00
ridiculousfish
3d40292c00 Switch env_var to using maybe_t
This eliminates the "missing" notion of env_var_t. Instead
env_get returns a maybe_t<env_var_t>, which forces callers to
handle the possibility that the variable is missing.
2017-09-01 00:14:42 -07:00
Kurtis Rader
f872f25f5b change env_var_t to a vector of strings
Internally fish should store vars as a vector of elements. The current
flat string representation is a holdover from when the code was written
in C.

Fixes #4200
2017-08-18 16:24:30 -07:00