Inner for loop when run in background in bash spawns new bash processbash variables in for loop rangeExplanation for background tasks on bashInner loop conditions shall depend on outer loop runfor loop in bashSkip for loop iteration depending on response to inner selectSafely kill foreground process when a background process endsbash: how for loop sorts files when iteratingrun another process after the background process is completeBash - Execute background process whilst reading outputCan't run a background process and other command from bash in one line: unxepected token `;'

Help to reproduce a tcolorbox with a decoration

Do I have to worry about players making “bad” choices on level up?

How to verbalise code in Mathematica?

What makes accurate emulation of old systems a difficult task?

Binary Numbers Magic Trick

Controversial area of mathematics

Error message with tabularx

Fizzy, soft, pop and still drinks

What is the relationship between spectral sequences and obstruction theory?

How can the Zone of Truth spell be defeated without the caster knowing?

Any examples of headwear for races with animal ears?

How can governments justify taking income tax on earnings, and then taking more tax on spending?

Hilbert Space and Banach Space

How do Bards prepare spells?

Was there a shared-world project before "Thieves World"?

How can I practically buy stocks?

How do we know that ממחרת השבת means from the first day of pesach and not the seventh?

How would one muzzle a full grown polar bear in the 13th century?

US visa is under administrative processing, I need the passport back ASAP

Why do Computer Science majors learn Calculus?

Sci-fi novel series with instant travel between planets through gates. A river runs through the gates

Mac Pro install disk keeps ejecting itself

simple conditions equation

How can Republicans who favour free markets, consistently express anger when they don't like the outcome of that choice?



Inner for loop when run in background in bash spawns new bash process


bash variables in for loop rangeExplanation for background tasks on bashInner loop conditions shall depend on outer loop runfor loop in bashSkip for loop iteration depending on response to inner selectSafely kill foreground process when a background process endsbash: how for loop sorts files when iteratingrun another process after the background process is completeBash - Execute background process whilst reading outputCan't run a background process and other command from bash in one line: unxepected token `;'






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;








1















I am executing below script



LOGDIR=~/curl_result_$(date |tr ' :' '_')

mkdir $LOGDIR

for THREADNO in $(seq 20)
do
for REQNO in $(seq 20)
do
time curl --verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii $LOGDIR/trace_$THREADNO_$REQNO -d @- <<EOF >> $LOGDIR/response_$THREADNO 2>&1
bc)"
EOF
echo -e "n-------------------------------" >> $LOGDIR/response_$THREADNO
done 2>&1 | grep real > $LOGDIR/timing_$THREADNO &
done


After sometime if i check for no of bash processes, it shows 20(not 1 or 21)



ps|grep bash|wc -l


The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned.
I want to avoid creating new shells as the CPU usage nears 100%.
I don't know if it matters, but i am using Cygwin.










share|improve this question






























    1















    I am executing below script



    LOGDIR=~/curl_result_$(date |tr ' :' '_')

    mkdir $LOGDIR

    for THREADNO in $(seq 20)
    do
    for REQNO in $(seq 20)
    do
    time curl --verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii $LOGDIR/trace_$THREADNO_$REQNO -d @- <<EOF >> $LOGDIR/response_$THREADNO 2>&1
    bc)"
    EOF
    echo -e "n-------------------------------" >> $LOGDIR/response_$THREADNO
    done 2>&1 | grep real > $LOGDIR/timing_$THREADNO &
    done


    After sometime if i check for no of bash processes, it shows 20(not 1 or 21)



    ps|grep bash|wc -l


    The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned.
    I want to avoid creating new shells as the CPU usage nears 100%.
    I don't know if it matters, but i am using Cygwin.










    share|improve this question


























      1












      1








      1








      I am executing below script



      LOGDIR=~/curl_result_$(date |tr ' :' '_')

      mkdir $LOGDIR

      for THREADNO in $(seq 20)
      do
      for REQNO in $(seq 20)
      do
      time curl --verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii $LOGDIR/trace_$THREADNO_$REQNO -d @- <<EOF >> $LOGDIR/response_$THREADNO 2>&1
      bc)"
      EOF
      echo -e "n-------------------------------" >> $LOGDIR/response_$THREADNO
      done 2>&1 | grep real > $LOGDIR/timing_$THREADNO &
      done


      After sometime if i check for no of bash processes, it shows 20(not 1 or 21)



      ps|grep bash|wc -l


      The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned.
      I want to avoid creating new shells as the CPU usage nears 100%.
      I don't know if it matters, but i am using Cygwin.










      share|improve this question
















      I am executing below script



      LOGDIR=~/curl_result_$(date |tr ' :' '_')

      mkdir $LOGDIR

      for THREADNO in $(seq 20)
      do
      for REQNO in $(seq 20)
      do
      time curl --verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii $LOGDIR/trace_$THREADNO_$REQNO -d @- <<EOF >> $LOGDIR/response_$THREADNO 2>&1
      bc)"
      EOF
      echo -e "n-------------------------------" >> $LOGDIR/response_$THREADNO
      done 2>&1 | grep real > $LOGDIR/timing_$THREADNO &
      done


      After sometime if i check for no of bash processes, it shows 20(not 1 or 21)



      ps|grep bash|wc -l


      The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned.
      I want to avoid creating new shells as the CPU usage nears 100%.
      I don't know if it matters, but i am using Cygwin.







      bash pipe background-process for






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited 55 mins ago









      Michael Homer

      51.6k8143180




      51.6k8143180










      asked 1 hour ago









      Manohar BhatManohar Bhat

      182




      182




















          2 Answers
          2






          active

          oldest

          votes


















          2














          Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:




          Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)




          It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.



          There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.



          If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.






          share|improve this answer






























            0














            What won't work



            Bash does not support threading parallelism, only multi-process parallelism.



            Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.



            I know nothing about Cygwin.




            Alternatives



            If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.



            Here's your your code, rewritten and tested:



            from datetime import datetime
            import json
            import random
            from plumbum import cmd as c
            from threading import Thread

            def now():
            return datetime.now().strftime("%Y-%m-%d %H_%M_%S")

            logdir = f"~/curl_result_now()"

            def curl(threadno, reqno):
            args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
            args2 = [f"logdir_threadno_reqno", "-d", "@-"]
            content = json.dumps(
            "name": f"logdir/trace_threadno_reqno_now()",
            "salary": random.randrange(100_000),
            "age": random.randrange(100_000) ,
            )
            call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
            print(call)
            # call()

            def curl_batch(threadno):
            for reqno in range(20):
            curl(threadno, reqno)

            # Start 20 threads
            threadList = []
            for threadno in range(20):
            t = Thread(target=curl_batch, args=(threadno,))
            t.start()
            threadList.append(t)

            # Wait for every thread
            for thread in threadList:
            thread.join()


            Enjoy Python's flexibilty ;)






            share|improve this answer

























              Your Answer








              StackExchange.ready(function()
              var channelOptions =
              tags: "".split(" "),
              id: "106"
              ;
              initTagRenderer("".split(" "), "".split(" "), channelOptions);

              StackExchange.using("externalEditor", function()
              // Have to fire editor after snippets, if snippets enabled
              if (StackExchange.settings.snippets.snippetsEnabled)
              StackExchange.using("snippets", function()
              createEditor();
              );

              else
              createEditor();

              );

              function createEditor()
              StackExchange.prepareEditor(
              heartbeatType: 'answer',
              autoActivateHeartbeat: false,
              convertImagesToLinks: false,
              noModals: true,
              showLowRepImageUploadWarning: true,
              reputationToPostImages: null,
              bindNavPrevention: true,
              postfix: "",
              imageUploader:
              brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
              contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
              allowUrls: true
              ,
              onDemand: true,
              discardSelector: ".discard-answer"
              ,immediatelyShowMarkdownHelp:true
              );



              );













              draft saved

              draft discarded


















              StackExchange.ready(
              function ()
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f515967%2finner-for-loop-when-run-in-background-in-bash-spawns-new-bash-process%23new-answer', 'question_page');

              );

              Post as a guest















              Required, but never shown

























              2 Answers
              2






              active

              oldest

              votes








              2 Answers
              2






              active

              oldest

              votes









              active

              oldest

              votes






              active

              oldest

              votes









              2














              Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:




              Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)




              It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.



              There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.



              If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.






              share|improve this answer



























                2














                Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:




                Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)




                It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.



                There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.



                If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.






                share|improve this answer

























                  2












                  2








                  2







                  Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:




                  Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)




                  It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.



                  There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.



                  If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.






                  share|improve this answer













                  Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:




                  Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)




                  It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.



                  There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.



                  If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered 55 mins ago









                  Michael HomerMichael Homer

                  51.6k8143180




                  51.6k8143180























                      0














                      What won't work



                      Bash does not support threading parallelism, only multi-process parallelism.



                      Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.



                      I know nothing about Cygwin.




                      Alternatives



                      If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.



                      Here's your your code, rewritten and tested:



                      from datetime import datetime
                      import json
                      import random
                      from plumbum import cmd as c
                      from threading import Thread

                      def now():
                      return datetime.now().strftime("%Y-%m-%d %H_%M_%S")

                      logdir = f"~/curl_result_now()"

                      def curl(threadno, reqno):
                      args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
                      args2 = [f"logdir_threadno_reqno", "-d", "@-"]
                      content = json.dumps(
                      "name": f"logdir/trace_threadno_reqno_now()",
                      "salary": random.randrange(100_000),
                      "age": random.randrange(100_000) ,
                      )
                      call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
                      print(call)
                      # call()

                      def curl_batch(threadno):
                      for reqno in range(20):
                      curl(threadno, reqno)

                      # Start 20 threads
                      threadList = []
                      for threadno in range(20):
                      t = Thread(target=curl_batch, args=(threadno,))
                      t.start()
                      threadList.append(t)

                      # Wait for every thread
                      for thread in threadList:
                      thread.join()


                      Enjoy Python's flexibilty ;)






                      share|improve this answer





























                        0














                        What won't work



                        Bash does not support threading parallelism, only multi-process parallelism.



                        Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.



                        I know nothing about Cygwin.




                        Alternatives



                        If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.



                        Here's your your code, rewritten and tested:



                        from datetime import datetime
                        import json
                        import random
                        from plumbum import cmd as c
                        from threading import Thread

                        def now():
                        return datetime.now().strftime("%Y-%m-%d %H_%M_%S")

                        logdir = f"~/curl_result_now()"

                        def curl(threadno, reqno):
                        args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
                        args2 = [f"logdir_threadno_reqno", "-d", "@-"]
                        content = json.dumps(
                        "name": f"logdir/trace_threadno_reqno_now()",
                        "salary": random.randrange(100_000),
                        "age": random.randrange(100_000) ,
                        )
                        call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
                        print(call)
                        # call()

                        def curl_batch(threadno):
                        for reqno in range(20):
                        curl(threadno, reqno)

                        # Start 20 threads
                        threadList = []
                        for threadno in range(20):
                        t = Thread(target=curl_batch, args=(threadno,))
                        t.start()
                        threadList.append(t)

                        # Wait for every thread
                        for thread in threadList:
                        thread.join()


                        Enjoy Python's flexibilty ;)






                        share|improve this answer



























                          0












                          0








                          0







                          What won't work



                          Bash does not support threading parallelism, only multi-process parallelism.



                          Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.



                          I know nothing about Cygwin.




                          Alternatives



                          If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.



                          Here's your your code, rewritten and tested:



                          from datetime import datetime
                          import json
                          import random
                          from plumbum import cmd as c
                          from threading import Thread

                          def now():
                          return datetime.now().strftime("%Y-%m-%d %H_%M_%S")

                          logdir = f"~/curl_result_now()"

                          def curl(threadno, reqno):
                          args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
                          args2 = [f"logdir_threadno_reqno", "-d", "@-"]
                          content = json.dumps(
                          "name": f"logdir/trace_threadno_reqno_now()",
                          "salary": random.randrange(100_000),
                          "age": random.randrange(100_000) ,
                          )
                          call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
                          print(call)
                          # call()

                          def curl_batch(threadno):
                          for reqno in range(20):
                          curl(threadno, reqno)

                          # Start 20 threads
                          threadList = []
                          for threadno in range(20):
                          t = Thread(target=curl_batch, args=(threadno,))
                          t.start()
                          threadList.append(t)

                          # Wait for every thread
                          for thread in threadList:
                          thread.join()


                          Enjoy Python's flexibilty ;)






                          share|improve this answer















                          What won't work



                          Bash does not support threading parallelism, only multi-process parallelism.



                          Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.



                          I know nothing about Cygwin.




                          Alternatives



                          If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.



                          Here's your your code, rewritten and tested:



                          from datetime import datetime
                          import json
                          import random
                          from plumbum import cmd as c
                          from threading import Thread

                          def now():
                          return datetime.now().strftime("%Y-%m-%d %H_%M_%S")

                          logdir = f"~/curl_result_now()"

                          def curl(threadno, reqno):
                          args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
                          args2 = [f"logdir_threadno_reqno", "-d", "@-"]
                          content = json.dumps(
                          "name": f"logdir/trace_threadno_reqno_now()",
                          "salary": random.randrange(100_000),
                          "age": random.randrange(100_000) ,
                          )
                          call = c.echo[content] | c.curl[(*args1, *args2)] >> f"logdir/response_threadno"
                          print(call)
                          # call()

                          def curl_batch(threadno):
                          for reqno in range(20):
                          curl(threadno, reqno)

                          # Start 20 threads
                          threadList = []
                          for threadno in range(20):
                          t = Thread(target=curl_batch, args=(threadno,))
                          t.start()
                          threadList.append(t)

                          # Wait for every thread
                          for thread in threadList:
                          thread.join()


                          Enjoy Python's flexibilty ;)







                          share|improve this answer














                          share|improve this answer



                          share|improve this answer








                          edited 22 mins ago

























                          answered 37 mins ago









                          Mathieu CAROFFMathieu CAROFF

                          2235




                          2235



























                              draft saved

                              draft discarded
















































                              Thanks for contributing an answer to Unix & Linux Stack Exchange!


                              • Please be sure to answer the question. Provide details and share your research!

                              But avoid


                              • Asking for help, clarification, or responding to other answers.

                              • Making statements based on opinion; back them up with references or personal experience.

                              To learn more, see our tips on writing great answers.




                              draft saved


                              draft discarded














                              StackExchange.ready(
                              function ()
                              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f515967%2finner-for-loop-when-run-in-background-in-bash-spawns-new-bash-process%23new-answer', 'question_page');

                              );

                              Post as a guest















                              Required, but never shown





















































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown

































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown







                              Popular posts from this blog

                              Era Viking Índice Início da Era Viquingue | Cotidiano | Sociedade | Língua | Religião | A arte | As primeiras cidades | As viagens dos viquingues | Viquingues do Oeste e Leste | Fim da Era Viquingue | Fontes históricas | Referências Bibliografia | Ligações externas | Menu de navegação«Sverige då!»«Handel I vikingetid»«O que é Nórdico Antigo»Mito, magia e religião na volsunga saga Um olhar sobre a trajetória mítica do herói sigurd«Bonden var den verklige vikingen»«Vikingatiden»«Vikingatiden»«Vinland»«Guerreiras de Óðinn: As Valkyrjor na Mitologia Viking»1519-9053«Esculpindo símbolos e seres: A arte viking em pedras rúnicas»1679-9313Historia - Tema: VikingarnaAventura e Magia no Mundo das Sagas IslandesasEra Vikinge

                              What's the metal clinking sound at the end of credits in Avengers: Endgame?What makes Thanos so strong in Avengers: Endgame?Who is the character that appears at the end of Endgame?What happens to Mjolnir (Thor's hammer) at the end of Endgame?The People's Ages in Avengers: EndgameWhat did Nebula do in Avengers: Endgame?Messing with time in the Avengers: Endgame climaxAvengers: Endgame timelineWhat are the time-travel rules in Avengers Endgame?Why use this song in Avengers: Endgame Opening Logo Sequence?Peggy's age in Avengers Endgame

                              Are there legal definitions of ethnicities/races? The 2019 Stack Overflow Developer Survey Results Are In Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Legal definitions in the United StatesAre there truly legal limits on US interest rates?Are gender identity and sexual orientation federally protected?Why is there an apparent legal bias against digital services?What limits are there to the powers of individual judges in the United States legal system?Are women only scholarships legal under Irish / EU law?Is the term “race” defined by Public Law enacted by Congress of the United StatesIs there a legal definition of race in the US?Neighbors are spying for landlord on Renters is it legal?Are Protected Classes Bi-directional?