Goroutines

In AWS you deal with multiple AWS accounts and multiple AWS regions. So often the requirement is to call an AWS multiple times. If you wait after each call, this really could take some time.

The good thing is, that AWS itself -with a few exceptions - can handle massive parallel calls to the API.

Parallel execution of call to regions: awsummary

The UseCase is to query all instances in all regions.

The API call to get the list of instances is ec2.DescribeInstances.

In sumec2/ec2.go we have:

  1. Create a configuration (line 29)

  2. Create a service specific client (line 34)

    19 func init() {
20   autoinit := awsummary.Autoinit()
21   if autoinit {
22     cfg, err := config.LoadDefaultConfig(context.TODO())
23     if err != nil {
24       panic("configuration error, " + err.Error())
25     }
26
27     Client = ec2.NewFromConfig(cfg)
28   }
29 }

  3. Prepare service specific input parameter

  4. Call the service

    61   resp, err := Client.DescribeInstances(context.TODO(), nil,
62   func(o *ec2.Options) {
63     o.Region = region
64   } )

    DescribeInstances is defined as:

        DescribeInstances(ctx context.Context,
    params *ec2.DescribeInstancesInput, optFns ...func(*ec2.Options)) (*ec2.DescribeInstancesOutput,
    error)

    In line 62-64 you see the use of the optFns to choose the region.

  5. Process results

    Now the response is iterated:

    70   for idx := range resp.Reservations {
71     for _, inst := range resp.Reservations[idx].Instances {

See Chapter Steps for using the SDK for a detailed explanation.

Call goroutines in parallel

I am using “github.com/panjf2000/ants/v2” as an framework for reusing goroutines, the main process stays the same:

  1. Create a WaitGroup:

    var wg sync.WaitGroup

    Here you store how many goroutines are running, so that you can wait until all executions are done

  2. Increment the waitgroup

    wg.Add(1)

  3. Create a goroutine

    go func() {
    defer wg.Done()
    // do something
}

    Defer makes sure that wg.Done() is called at the end of the func()

  4. Wait until all parallel executions are done:

        wg.Wait()

In the awsummary app this is done in main/main.go

 86       for _, region := range regions {
 87         region := region
 88
 89         calcInstances := func() {
 90           defer wg.Done()
 91           // Getting EC2 data
 92           instancesStats := sumec2.ListInstances(region, verbose)
 93           if instancesStats.Total > 0 {
 94             logs.WithFields(logs.Fields{
 95               "EC2":               instancesStats.Total,
 96               "EC2Running":        instancesStats.Running,
 97               "EC2RunningWindows": instancesStats.Windows,
 98               "Region":            region,
 99             }).Info(awsummary.Msg("EC2"))
100           }
101           totalInstancesStats.Total += instancesStats.Total
102           totalInstancesStats.Running += instancesStats.Running
103           totalInstancesStats.Windows += instancesStats.Windows
104         }
105         wg.Add(1)
106         _ = pool.Submit(calcInstances)

The ants tool reuses goroutines, which speeds up the app a little bit more.

And near the end the app waits for all goroutines:

242       wg.Wait()

Source

See also