Inner for loop when run in background in bash spawns new bash processbash variables in for loop rangeExplanation for background tasks on bashInner loop conditions shall depend on outer loop runfor loop in bashSkip for loop iteration depending on response to inner selectSafely kill foreground process when a background process endsbash: how for loop sorts files when iteratingrun another process after the background process is completeBash - Execute background process whilst reading outputCan't run a background process and other command from bash in one line: unxepected token `;'
Help to reproduce a tcolorbox with a decoration
Do I have to worry about players making “bad” choices on level up?
How to verbalise code in Mathematica?
What makes accurate emulation of old systems a difficult task?
Binary Numbers Magic Trick
Controversial area of mathematics
Error message with tabularx
Fizzy, soft, pop and still drinks
What is the relationship between spectral sequences and obstruction theory?
How can the Zone of Truth spell be defeated without the caster knowing?
Any examples of headwear for races with animal ears?
How can governments justify taking income tax on earnings, and then taking more tax on spending?
Hilbert Space and Banach Space
How do Bards prepare spells?
Was there a shared-world project before "Thieves World"?
How can I practically buy stocks?
How do we know that ממחרת השבת means from the first day of pesach and not the seventh?
How would one muzzle a full grown polar bear in the 13th century?
US visa is under administrative processing, I need the passport back ASAP
Why do Computer Science majors learn Calculus?
Sci-fi novel series with instant travel between planets through gates. A river runs through the gates
Mac Pro install disk keeps ejecting itself
simple conditions equation
How can Republicans who favour free markets, consistently express anger when they don't like the outcome of that choice?
Inner for loop when run in background in bash spawns new bash process
bash variables in for loop rangeExplanation for background tasks on bashInner loop conditions shall depend on outer loop runfor loop in bashSkip for loop iteration depending on response to inner selectSafely kill foreground process when a background process endsbash: how for loop sorts files when iteratingrun another process after the background process is completeBash - Execute background process whilst reading outputCan't run a background process and other command from bash in one line: unxepected token `;'
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I am executing below script
LOGDIR=~/curl_result_$(date |tr ' :' '_')
mkdir $LOGDIR
for THREADNO in $(seq 20)
do
for REQNO in $(seq 20)
do
time curl --verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii $LOGDIR/trace_$THREADNO_$REQNO -d @- <<EOF >> $LOGDIR/response_$THREADNO 2>&1
bc)"
EOF
echo -e "n-------------------------------" >> $LOGDIR/response_$THREADNO
done 2>&1 | grep real > $LOGDIR/timing_$THREADNO &
done
After sometime if i check for no of bash processes, it shows 20(not 1 or 21)
ps|grep bash|wc -l
The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned.
I want to avoid creating new shells as the CPU usage nears 100%.
I don't know if it matters, but i am using Cygwin.
bash pipe background-process for
add a comment |
I am executing below script
LOGDIR=~/curl_result_$(date |tr ' :' '_')
mkdir $LOGDIR
for THREADNO in $(seq 20)
do
for REQNO in $(seq 20)
do
time curl --verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii $LOGDIR/trace_$THREADNO_$REQNO -d @- <<EOF >> $LOGDIR/response_$THREADNO 2>&1
bc)"
EOF
echo -e "n-------------------------------" >> $LOGDIR/response_$THREADNO
done 2>&1 | grep real > $LOGDIR/timing_$THREADNO &
done
After sometime if i check for no of bash processes, it shows 20(not 1 or 21)
ps|grep bash|wc -l
The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned.
I want to avoid creating new shells as the CPU usage nears 100%.
I don't know if it matters, but i am using Cygwin.
bash pipe background-process for
add a comment |
I am executing below script
LOGDIR=~/curl_result_$(date |tr ' :' '_')
mkdir $LOGDIR
for THREADNO in $(seq 20)
do
for REQNO in $(seq 20)
do
time curl --verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii $LOGDIR/trace_$THREADNO_$REQNO -d @- <<EOF >> $LOGDIR/response_$THREADNO 2>&1
bc)"
EOF
echo -e "n-------------------------------" >> $LOGDIR/response_$THREADNO
done 2>&1 | grep real > $LOGDIR/timing_$THREADNO &
done
After sometime if i check for no of bash processes, it shows 20(not 1 or 21)
ps|grep bash|wc -l
The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned.
I want to avoid creating new shells as the CPU usage nears 100%.
I don't know if it matters, but i am using Cygwin.
bash pipe background-process for
I am executing below script
LOGDIR=~/curl_result_$(date |tr ' :' '_')
mkdir $LOGDIR
for THREADNO in $(seq 20)
do
for REQNO in $(seq 20)
do
time curl --verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii $LOGDIR/trace_$THREADNO_$REQNO -d @- <<EOF >> $LOGDIR/response_$THREADNO 2>&1
bc)"
EOF
echo -e "n-------------------------------" >> $LOGDIR/response_$THREADNO
done 2>&1 | grep real > $LOGDIR/timing_$THREADNO &
done
After sometime if i check for no of bash processes, it shows 20(not 1 or 21)
ps|grep bash|wc -l
The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned.
I want to avoid creating new shells as the CPU usage nears 100%.
I don't know if it matters, but i am using Cygwin.
bash pipe background-process for
bash pipe background-process for
edited 55 mins ago
Michael Homer
51.6k8143180
51.6k8143180
asked 1 hour ago
Manohar BhatManohar Bhat
182
182
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:
Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)
It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.
There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.
If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.
add a comment |
What won't work
Bash does not support threading parallelism, only multi-process parallelism.
Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.
I know nothing about Cygwin.
Alternatives
If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.
Here's your your code, rewritten and tested:
from datetime import datetime
import json
import random
from plumbum import cmd as c
from threading import Thread
def now():
return datetime.now().strftime("%Y-%m-%d %H_%M_%S")
logdir = f"~/curl_result_now()"
def curl(threadno, reqno):
args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
args2 = [f"logdir_threadno_reqno", "-d", "@-"]
content = json.dumps(
"name": f"logdir/trace_threadno_reqno_now()",
"salary": random.randrange(100_000),
"age": random.randrange(100_000) ,
)
call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
print(call)
# call()
def curl_batch(threadno):
for reqno in range(20):
curl(threadno, reqno)
# Start 20 threads
threadList = []
for threadno in range(20):
t = Thread(target=curl_batch, args=(threadno,))
t.start()
threadList.append(t)
# Wait for every thread
for thread in threadList:
thread.join()
Enjoy Python's flexibilty ;)
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f515967%2finner-for-loop-when-run-in-background-in-bash-spawns-new-bash-process%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:
Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)
It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.
There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.
If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.
add a comment |
Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:
Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)
It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.
There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.
If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.
add a comment |
Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:
Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)
It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.
There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.
If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.
Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:
Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)
It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.
There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.
If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.
answered 55 mins ago
Michael HomerMichael Homer
51.6k8143180
51.6k8143180
add a comment |
add a comment |
What won't work
Bash does not support threading parallelism, only multi-process parallelism.
Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.
I know nothing about Cygwin.
Alternatives
If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.
Here's your your code, rewritten and tested:
from datetime import datetime
import json
import random
from plumbum import cmd as c
from threading import Thread
def now():
return datetime.now().strftime("%Y-%m-%d %H_%M_%S")
logdir = f"~/curl_result_now()"
def curl(threadno, reqno):
args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
args2 = [f"logdir_threadno_reqno", "-d", "@-"]
content = json.dumps(
"name": f"logdir/trace_threadno_reqno_now()",
"salary": random.randrange(100_000),
"age": random.randrange(100_000) ,
)
call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
print(call)
# call()
def curl_batch(threadno):
for reqno in range(20):
curl(threadno, reqno)
# Start 20 threads
threadList = []
for threadno in range(20):
t = Thread(target=curl_batch, args=(threadno,))
t.start()
threadList.append(t)
# Wait for every thread
for thread in threadList:
thread.join()
Enjoy Python's flexibilty ;)
add a comment |
What won't work
Bash does not support threading parallelism, only multi-process parallelism.
Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.
I know nothing about Cygwin.
Alternatives
If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.
Here's your your code, rewritten and tested:
from datetime import datetime
import json
import random
from plumbum import cmd as c
from threading import Thread
def now():
return datetime.now().strftime("%Y-%m-%d %H_%M_%S")
logdir = f"~/curl_result_now()"
def curl(threadno, reqno):
args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
args2 = [f"logdir_threadno_reqno", "-d", "@-"]
content = json.dumps(
"name": f"logdir/trace_threadno_reqno_now()",
"salary": random.randrange(100_000),
"age": random.randrange(100_000) ,
)
call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
print(call)
# call()
def curl_batch(threadno):
for reqno in range(20):
curl(threadno, reqno)
# Start 20 threads
threadList = []
for threadno in range(20):
t = Thread(target=curl_batch, args=(threadno,))
t.start()
threadList.append(t)
# Wait for every thread
for thread in threadList:
thread.join()
Enjoy Python's flexibilty ;)
add a comment |
What won't work
Bash does not support threading parallelism, only multi-process parallelism.
Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.
I know nothing about Cygwin.
Alternatives
If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.
Here's your your code, rewritten and tested:
from datetime import datetime
import json
import random
from plumbum import cmd as c
from threading import Thread
def now():
return datetime.now().strftime("%Y-%m-%d %H_%M_%S")
logdir = f"~/curl_result_now()"
def curl(threadno, reqno):
args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
args2 = [f"logdir_threadno_reqno", "-d", "@-"]
content = json.dumps(
"name": f"logdir/trace_threadno_reqno_now()",
"salary": random.randrange(100_000),
"age": random.randrange(100_000) ,
)
call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
print(call)
# call()
def curl_batch(threadno):
for reqno in range(20):
curl(threadno, reqno)
# Start 20 threads
threadList = []
for threadno in range(20):
t = Thread(target=curl_batch, args=(threadno,))
t.start()
threadList.append(t)
# Wait for every thread
for thread in threadList:
thread.join()
Enjoy Python's flexibilty ;)
What won't work
Bash does not support threading parallelism, only multi-process parallelism.
Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.
I know nothing about Cygwin.
Alternatives
If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.
Here's your your code, rewritten and tested:
from datetime import datetime
import json
import random
from plumbum import cmd as c
from threading import Thread
def now():
return datetime.now().strftime("%Y-%m-%d %H_%M_%S")
logdir = f"~/curl_result_now()"
def curl(threadno, reqno):
args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
args2 = [f"logdir_threadno_reqno", "-d", "@-"]
content = json.dumps(
"name": f"logdir/trace_threadno_reqno_now()",
"salary": random.randrange(100_000),
"age": random.randrange(100_000) ,
)
call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
print(call)
# call()
def curl_batch(threadno):
for reqno in range(20):
curl(threadno, reqno)
# Start 20 threads
threadList = []
for threadno in range(20):
t = Thread(target=curl_batch, args=(threadno,))
t.start()
threadList.append(t)
# Wait for every thread
for thread in threadList:
thread.join()
Enjoy Python's flexibilty ;)
edited 22 mins ago
answered 37 mins ago
Mathieu CAROFFMathieu CAROFF
2235
2235
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f515967%2finner-for-loop-when-run-in-background-in-bash-spawns-new-bash-process%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown